Nov 01 03:39:13.447208 ip-10-0-128-69 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 03:39:13.447218 ip-10-0-128-69 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 03:39:13.447227 ip-10-0-128-69 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 03:39:13.447538 ip-10-0-128-69 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 03:39:23.696845 ip-10-0-128-69 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 03:39:23.696880 ip-10-0-128-69 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3fdf7361e8a140cebb8df36c0812d938 -- Nov 01 03:41:51.920926 ip-10-0-128-69 systemd[1]: Starting Kubernetes Kubelet... Nov 01 03:41:52.377324 ip-10-0-128-69 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 03:41:52.377324 ip-10-0-128-69 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 03:41:52.377324 ip-10-0-128-69 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 03:41:52.377324 ip-10-0-128-69 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 03:41:52.381227 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381060 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.381227 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381228 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381234 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381238 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381242 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381245 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381248 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381251 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381254 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381257 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381259 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381262 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381264 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381267 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381270 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381276 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381279 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381282 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381284 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381287 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381290 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.381298 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381292 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381295 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381297 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381300 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381303 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381305 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381308 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381311 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381313 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381316 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381318 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381321 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381324 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381326 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381329 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381332 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381334 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381336 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381339 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.381770 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381341 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381344 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381346 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381349 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381351 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381353 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381356 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381359 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381362 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381364 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381367 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381369 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381372 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381374 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381376 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381379 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381382 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381384 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381387 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381389 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.382209 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381392 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381396 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381400 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381402 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381412 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381415 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381418 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381420 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381423 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381425 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381428 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381430 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381433 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381436 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381438 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381441 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381443 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381446 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381448 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.382865 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381451 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381454 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381456 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381459 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381461 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381464 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381466 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381469 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381473 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381475 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381478 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381480 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381483 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381486 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381496 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381499 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381502 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381504 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381507 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.383519 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381509 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381512 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381515 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381517 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381520 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381522 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381524 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.381527 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382491 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382504 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382509 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382514 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382518 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382524 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382528 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382544 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382549 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382553 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382557 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.383994 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382562 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382572 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382594 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382599 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382603 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382608 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382612 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382616 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382629 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382634 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382638 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382642 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382646 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382650 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382660 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382665 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382669 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382673 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382678 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382682 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.384438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382686 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382690 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382694 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382698 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382702 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382707 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382717 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382721 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382726 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382731 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382736 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382740 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382744 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382748 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382752 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382756 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382760 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382764 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382775 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.384925 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382784 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382788 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382792 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382796 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382800 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382804 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382808 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382812 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382816 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382820 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382824 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382833 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382838 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382842 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382846 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382850 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382855 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382859 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382864 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382869 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.385392 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382873 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382877 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382882 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382887 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382896 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382900 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382907 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382917 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382921 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382925 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382929 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382933 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382937 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382941 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382946 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382955 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382959 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382963 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.385876 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382967 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382976 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382982 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382988 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382993 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.382998 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383002 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383007 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383011 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383019 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383024 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383029 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383033 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383037 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383041 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.383045 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384051 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384063 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384067 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.386320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384070 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384073 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384076 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384078 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384081 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384084 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384086 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384089 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384092 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384094 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384097 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384099 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384102 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384104 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384107 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384109 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384112 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384114 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384117 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384120 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.386801 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384123 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384127 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384131 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384133 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384136 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384141 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384145 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384148 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384151 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384154 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384156 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384159 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384162 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384164 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384167 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384170 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384172 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384174 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384177 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.387282 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384180 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384182 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384185 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384187 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384190 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384193 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384195 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384198 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384201 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384204 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384206 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384212 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384214 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384216 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384220 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384222 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384224 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384227 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384230 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.387741 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384232 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384235 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384238 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384240 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384243 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384246 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384248 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384251 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384253 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384256 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384258 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384260 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384263 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384265 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384267 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384270 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384272 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384275 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384278 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.388249 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384281 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384284 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384286 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384289 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384292 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384294 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384297 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384299 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384302 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384304 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384306 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384309 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384311 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384314 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384317 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384319 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384322 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384324 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384327 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.388711 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384329 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384332 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384335 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384337 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384339 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.384343 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384443 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384450 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384457 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384462 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384467 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384470 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384475 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384479 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384482 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384486 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384489 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384492 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384495 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384499 2579 flags.go:64] FLAG: --cgroup-root="" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384502 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384505 2579 flags.go:64] FLAG: --client-ca-file="" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384507 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 01 03:41:52.389160 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384510 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384514 2579 flags.go:64] FLAG: --cluster-domain="" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384517 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384520 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384523 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384526 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384530 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384533 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384536 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384539 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384542 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384545 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384547 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384551 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384553 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384558 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384561 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384564 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384567 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384570 2579 flags.go:64] FLAG: --enable-server="true" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384573 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384595 2579 flags.go:64] FLAG: --event-burst="100" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384599 2579 flags.go:64] FLAG: --event-qps="50" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384602 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 03:41:52.389808 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384605 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384608 2579 flags.go:64] FLAG: --eviction-hard="" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384613 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384616 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384619 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384622 2579 flags.go:64] FLAG: --eviction-soft="" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384624 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384627 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384630 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384633 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384636 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384639 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384642 2579 flags.go:64] FLAG: --feature-gates="" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384646 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384649 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384652 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384655 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384658 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384661 2579 flags.go:64] FLAG: --help="false" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384664 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384667 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384670 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384673 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 03:41:52.390382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384681 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384685 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384688 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384690 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384694 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384697 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384700 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384703 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384706 2579 flags.go:64] FLAG: --kube-reserved="" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384709 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384712 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384715 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384718 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384721 2579 flags.go:64] FLAG: --lock-file="" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384724 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384727 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384730 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384740 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384743 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384746 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384748 2579 flags.go:64] FLAG: --logging-format="text" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384751 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384755 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384758 2579 flags.go:64] FLAG: --manifest-url="" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384760 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 01 03:41:52.390962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384765 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384768 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384772 2579 flags.go:64] FLAG: --max-pods="110" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384775 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384778 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384781 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384784 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384787 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384790 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384793 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384801 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384804 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384807 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384810 2579 flags.go:64] FLAG: --pod-cidr="" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384813 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384816 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384819 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384822 2579 flags.go:64] FLAG: --port="10250" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384825 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384828 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0cb39e7e1d45a7d12" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384831 2579 flags.go:64] FLAG: --qos-reserved="" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384834 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384837 2579 flags.go:64] FLAG: --register-node="true" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384839 2579 flags.go:64] FLAG: --register-with-taints="" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384843 2579 flags.go:64] FLAG: --registry-burst="10" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384846 2579 flags.go:64] FLAG: --registry-qps="5" Nov 01 03:41:52.391604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384848 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384851 2579 flags.go:64] FLAG: --reserved-memory="" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384855 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384858 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384861 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384864 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384867 2579 flags.go:64] FLAG: --runonce="false" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384870 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384873 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384876 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384879 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384881 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384885 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384888 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384891 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384896 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384899 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384902 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384905 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384909 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384914 2579 flags.go:64] FLAG: --system-cgroups="" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384917 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384922 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384925 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384928 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 03:41:52.392199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384932 2579 flags.go:64] FLAG: --tls-min-version="" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384935 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384938 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384941 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384944 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384947 2579 flags.go:64] FLAG: --v="2" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384951 2579 flags.go:64] FLAG: --version="false" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384956 2579 flags.go:64] FLAG: --vmodule="" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384960 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.384963 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385115 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385119 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385122 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385125 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385128 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385130 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385133 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385135 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385138 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385141 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385144 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.392805 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385146 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385151 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385153 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385156 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385158 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385161 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385166 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385168 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385171 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385174 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385176 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385179 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385181 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385184 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385186 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385189 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385191 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385193 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385196 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385198 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.393280 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385201 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385204 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385206 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385209 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385211 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385214 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385216 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385221 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385223 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385226 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385228 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385231 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385234 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385237 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385240 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385242 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385245 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385248 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385252 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.393768 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385254 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385257 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385259 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385262 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385264 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385266 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385269 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385271 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385274 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385276 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385279 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385281 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385284 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385286 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385289 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385291 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385294 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385296 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385298 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.394238 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385301 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385304 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385306 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385308 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385311 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385314 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385317 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385320 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385322 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385325 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385327 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385331 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385333 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385337 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385339 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385343 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385347 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385350 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385353 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.394746 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385357 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385360 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385363 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385365 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385368 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385370 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385373 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385375 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385377 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385380 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385382 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385385 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385387 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385390 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385393 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385396 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.395189 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385398 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.385424 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385460 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385465 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385470 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385473 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385476 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385479 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385483 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385486 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385489 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385492 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385494 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385497 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385500 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385502 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.395568 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385504 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385507 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385510 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385512 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385514 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385517 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385520 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385522 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385525 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385527 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385532 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385536 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385539 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385542 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385546 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385549 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385551 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385554 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385557 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.395958 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385560 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385563 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385566 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385568 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385571 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385593 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385598 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385601 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385604 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385610 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385613 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385616 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385618 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385621 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385623 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385626 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385628 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385631 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385633 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.396394 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385636 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385638 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385641 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385644 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385647 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385650 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385652 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385655 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385657 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385660 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385663 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385665 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385668 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385670 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385674 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385676 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385679 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385682 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385685 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385688 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.396846 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385690 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385693 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385696 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385700 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385703 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385705 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385708 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385710 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385713 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385715 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385718 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385720 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385723 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385725 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385727 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385730 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385732 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385735 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385737 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385740 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.397346 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385742 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385745 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385747 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385751 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385753 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385755 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385759 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385762 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385764 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385767 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385770 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385773 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385775 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.385778 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.392888 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.392905 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.392919 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.392923 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.392983 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.397818 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.392988 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.392991 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.392994 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.392997 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393000 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393003 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393005 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393008 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393010 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393013 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393015 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393018 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393021 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393023 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393026 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393029 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393032 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393034 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393036 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393039 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.398255 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393041 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393044 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393047 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393049 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393052 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393054 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393057 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393060 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393064 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393067 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393070 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393072 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393075 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393077 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393080 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393082 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393085 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393089 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.398742 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393093 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393096 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393099 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393101 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393104 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393107 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393109 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393111 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393114 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393116 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393119 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393121 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393124 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393126 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393129 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393131 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393134 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393136 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393139 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393141 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.399159 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393143 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393147 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393150 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393153 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393156 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393159 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393161 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393164 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393166 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393169 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393171 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393174 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393176 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393178 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393181 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393183 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393186 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393188 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393190 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393193 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.399686 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393195 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393198 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393200 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393202 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393205 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393207 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393209 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393212 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393214 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393216 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393219 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393222 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393224 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393227 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393229 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393232 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393236 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393240 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393243 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.400167 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393246 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393248 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393251 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393254 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393256 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393259 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393261 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393264 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.393293 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393456 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393461 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393464 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393467 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393470 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393474 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.400645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393477 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393480 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393483 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393486 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393489 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393492 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393494 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393497 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393500 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393505 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393508 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393511 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393514 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393517 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393520 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393523 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393526 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393528 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393531 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.401125 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393533 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393536 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393538 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393541 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393543 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393546 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393548 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393550 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393553 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393555 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393557 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393560 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393562 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393565 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393567 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393570 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393572 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393592 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393595 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393598 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.401754 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393600 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393603 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393605 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393607 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393610 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393612 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393615 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393618 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393620 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393623 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393626 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393629 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393631 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393634 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393636 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393638 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393641 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393643 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393645 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.402224 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393648 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393650 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393653 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393655 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393657 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393660 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393662 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393664 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393667 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393669 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393672 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393674 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393677 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393679 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393681 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393684 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393686 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393689 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393691 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393693 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.402691 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393696 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393699 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393701 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393703 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393706 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393709 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393711 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393714 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393716 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393718 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393721 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393723 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393725 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393728 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393730 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393733 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393735 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393737 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393740 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393742 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.403210 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393745 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393747 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.393771 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393807 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393812 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393818 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393820 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393822 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393825 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393828 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393830 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393833 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393835 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393837 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:41:52.403872 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393840 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393842 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393845 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393848 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393851 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393853 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393856 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393858 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393860 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393863 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393865 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393868 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393871 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393873 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393875 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393878 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393883 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393886 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:41:52.404266 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393888 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393891 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393893 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393895 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393898 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393900 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393903 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393906 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393908 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393911 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393913 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393915 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393918 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393920 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393923 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393925 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393931 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393934 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393936 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393939 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:41:52.404727 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393941 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393944 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393946 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393951 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393954 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393956 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393959 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393961 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393963 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393966 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393968 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393973 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393976 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393978 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393981 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393983 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393985 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393988 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:41:52.405179 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393991 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393993 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393995 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.393998 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394000 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394003 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394007 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394010 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394014 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394018 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394020 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394024 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394026 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394029 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394031 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394035 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394041 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394044 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:41:52.405735 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394047 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394049 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394052 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394054 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394056 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394059 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394061 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394064 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394067 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394069 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394075 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394077 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394082 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394085 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:52.394087 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.394136 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.396070 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.397021 2579 server.go:1014] "Starting client certificate rotation" Nov 01 03:41:52.406320 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.397120 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 03:41:52.406828 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.397156 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 03:41:52.406828 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.399434 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 03:41:52.406828 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.400791 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 03:41:52.413764 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.413747 2579 log.go:25] "Validated CRI v1 runtime API" Nov 01 03:41:52.418266 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.417825 2579 log.go:25] "Validated CRI v1 image API" Nov 01 03:41:52.419099 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.419072 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 03:41:52.421884 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.421865 2579 fs.go:135] Filesystem UUIDs: map[10401c2b-3bae-4dd2-86b7-509d4d73f4ef:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d5290e1b-521a-4723-8947-cc279d96ed3e:/dev/nvme0n1p4] Nov 01 03:41:52.421944 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.421884 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 03:41:52.429612 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.429482 2579 manager.go:217] Machine: {Timestamp:2026-11-01 03:41:52.427056245 +0000 UTC m=+0.376820946 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099490 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23408ad730643961cad07b37f22d67 SystemUUID:ec23408a-d730-6439-61ca-d07b37f22d67 BootID:3fdf7361-e8a1-40ce-bb8d-f36c0812d938 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c3:1e:9e:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c3:1e:9e:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:00:ed:fa:e3:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 03:41:52.430070 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.430060 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 03:41:52.430156 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.430144 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 03:41:52.433725 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.433701 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 03:41:52.433862 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.433727 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-69.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 03:41:52.433903 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.433865 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 03:41:52.433903 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.433873 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 03:41:52.433965 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.433905 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 03:41:52.433965 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.433924 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 03:41:52.434748 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.434736 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 03:41:52.438382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.438368 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 03:41:52.438501 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.438492 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 03:41:52.439256 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.439245 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 01 03:41:52.439312 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.439264 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 03:41:52.439312 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.439284 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 03:41:52.439312 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.439297 2579 kubelet.go:405] "Adding apiserver pod source" Nov 01 03:41:52.439312 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.439308 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 03:41:52.439715 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.439689 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 03:41:52.442470 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.442454 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 03:41:52.448163 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.448150 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 03:41:52.449358 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449345 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 03:41:52.449400 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449376 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 03:41:52.449533 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449523 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 03:41:52.449566 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449538 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 03:41:52.449566 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449545 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 03:41:52.449566 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449551 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 03:41:52.449566 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449558 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 03:41:52.449566 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.449564 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 03:41:52.450566 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.450557 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 03:41:52.450613 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.450567 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 03:41:52.450613 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.450574 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 03:41:52.450613 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.450596 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 03:41:52.450613 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.450609 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 03:41:52.450739 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.450628 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 03:41:52.451502 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.451494 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 03:41:52.451534 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.451503 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 03:41:52.455200 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.455189 2579 server.go:1263] "Started kubelet" Nov 01 03:41:52.455289 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.455264 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 03:41:52.455400 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.455337 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 03:41:52.455459 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.455430 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 03:41:52.456058 ip-10-0-128-69 systemd[1]: Started Kubernetes Kubelet. Nov 01 03:41:52.457097 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.457074 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 03:41:52.457768 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.457754 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 01 03:41:52.460920 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.460899 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 03:41:52.463377 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.463365 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 03:41:52.463440 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.463380 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 03:41:52.463988 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.463944 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:52.464803 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.464782 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 03:41:52.465153 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465135 2579 factory.go:55] Registering systemd factory Nov 01 03:41:52.465232 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465157 2579 factory.go:223] Registration of the systemd container factory successfully Nov 01 03:41:52.465307 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465287 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 03:41:52.465380 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465311 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 03:41:52.465486 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465470 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 01 03:41:52.465540 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465487 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 01 03:41:52.465645 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465628 2579 factory.go:153] Registering CRI-O factory Nov 01 03:41:52.465695 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465647 2579 factory.go:223] Registration of the crio container factory successfully Nov 01 03:41:52.465737 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465707 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 03:41:52.465780 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465738 2579 factory.go:103] Registering Raw factory Nov 01 03:41:52.465780 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.465766 2579 manager.go:1199] Started watching for new ooms in manager Nov 01 03:41:52.466733 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.466718 2579 manager.go:319] Starting recovery of all containers Nov 01 03:41:52.478628 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.478599 2579 manager.go:324] Recovery completed Nov 01 03:41:52.482834 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.482822 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:52.484652 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.484632 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:52.484731 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.484667 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:52.484731 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.484683 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:52.485216 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.485203 2579 cpu_manager.go:226] "Starting" policy="none" Nov 01 03:41:52.485289 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.485216 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 03:41:52.485289 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.485220 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:52.485289 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.485241 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 03:41:52.485955 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.485940 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:52.486037 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.485970 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:52.486037 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.485986 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:52.487331 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.487320 2579 policy_none.go:50] "Start" Nov 01 03:41:52.487385 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.487334 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 03:41:52.487385 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.487344 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 03:41:52.489365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.489350 2579 policy_none.go:44] "Start" Nov 01 03:41:52.528060 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.527884 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 03:41:52.539590 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.539566 2579 manager.go:341] "Starting Device Plugin manager" Nov 01 03:41:52.539675 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.539634 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 03:41:52.539675 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.539647 2579 server.go:87] "Starting device plugin registration server" Nov 01 03:41:52.539919 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.539903 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 03:41:52.539992 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.539921 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 03:41:52.540047 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.540027 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 03:41:52.540130 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.540113 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 03:41:52.540130 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.540126 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 03:41:52.540708 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.540685 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 03:41:52.540793 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.540746 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:52.589712 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.589678 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-69.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 03:41:52.589712 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.589695 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-69.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 03:41:52.591179 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.589675 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-69.us-west-2.compute.internal.18e3ceda07f27589 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 03:41:52.455112073 +0000 UTC m=+0.404876774,LastTimestamp:2026-11-01 03:41:52.455112073 +0000 UTC m=+0.404876774,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.us-west-2.compute.internal,}" Nov 01 03:41:52.604477 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.604450 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 03:41:52.605769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.605749 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 03:41:52.605769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.605771 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 03:41:52.605903 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.605799 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 03:41:52.605903 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.605838 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 03:41:52.640377 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.640098 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:52.640377 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.640082 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-69.us-west-2.compute.internal.18e3ceda09b53359 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-69.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 03:41:52.484651865 +0000 UTC m=+0.434416567,LastTimestamp:2026-11-01 03:41:52.484651865 +0000 UTC m=+0.434416567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.us-west-2.compute.internal,}" Nov 01 03:41:52.640377 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.640182 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 03:41:52.640377 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.640361 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-69.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 03:41:52.640698 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.640420 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 03:41:52.641487 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.641465 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:52.641620 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.641508 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:52.641620 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.641524 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:52.641620 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.641561 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.679788 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.679755 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 03:41:52.679916 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.679832 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwxnw" Nov 01 03:41:52.707006 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.706963 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal"] Nov 01 03:41:52.707119 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.707040 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:52.707971 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.707956 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:52.708051 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.707982 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:52.708051 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.707992 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:52.709667 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.709655 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:52.709838 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.709825 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.709875 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.709853 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:52.710360 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.710344 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:52.710360 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.710357 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:52.710469 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.710373 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:52.710469 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.710377 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:52.710469 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.710387 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:52.710469 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.710394 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:52.711756 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.711744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.711802 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.711767 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:52.712431 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.712410 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:52.712527 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.712442 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:52.712527 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.712452 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:52.731197 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.731174 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" node="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.734102 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.734084 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" node="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.740065 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.740052 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:41:52.745604 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.745588 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.745687 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.745609 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-69.us-west-2.compute.internal\": node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:52.766968 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.766939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.767074 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.766967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.767074 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.766987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63ad88aaffc4eb2dd4273e995e15f57f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"63ad88aaffc4eb2dd4273e995e15f57f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.857763 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.857731 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:52.868053 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.868031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.868139 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.868116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.868139 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.868128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.868226 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.868157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63ad88aaffc4eb2dd4273e995e15f57f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"63ad88aaffc4eb2dd4273e995e15f57f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.868226 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.868195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63ad88aaffc4eb2dd4273e995e15f57f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"63ad88aaffc4eb2dd4273e995e15f57f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.868287 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:52.868222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:52.958459 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:52.958393 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.035306 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.035272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:53.039211 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.039189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:53.059451 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.059429 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.160310 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.160278 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.260859 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.260780 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.361637 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.361601 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.397367 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.397337 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 03:41:53.397823 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.397526 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 03:41:53.397823 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.397505 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://afdb05243526b4c0e91ef2f637640017-e887bece4d7f5b9e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-69.us-west-2.compute.internal.18e3ceda09b5bff1\": read tcp 10.0.128.69:44584->3.135.252.134:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-69.us-west-2.compute.internal.18e3ceda09b5bff1 default 5436 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-69.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-69.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-69.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 03:41:52 +0000 UTC,LastTimestamp:2026-11-01 03:41:52.707996413 +0000 UTC m=+0.657761114,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-69.us-west-2.compute.internal,}" Nov 01 03:41:53.461707 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.461673 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.562080 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.562049 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.651798 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.651764 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:41:53.653944 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.653925 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:41:53.654052 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.653956 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:41:53.654052 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.653966 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:41:53.663087 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.663065 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.677668 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.677645 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:41:53.683741 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.683715 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:36:53 +0000 UTC" deadline="2028-04-09 09:43:57.237618293 +0000 UTC" Nov 01 03:41:53.683741 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.683738 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12606h2m3.553882386s" Nov 01 03:41:53.764011 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.763978 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.864272 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.864200 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.965212 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:53.965172 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:53.966345 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:53.966328 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:41:54.065646 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:54.065615 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:55.344565 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.344506 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:41:55.359122 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:55.359072 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ad88aaffc4eb2dd4273e995e15f57f.slice/crio-249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5 WatchSource:0}: Error finding container 249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5: Status 404 returned error can't find the container with id 249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5 Nov 01 03:41:55.359435 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:55.359412 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc61107e38660d4c05b158edffa7de6.slice/crio-d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a WatchSource:0}: Error finding container d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a: Status 404 returned error can't find the container with id d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a Nov 01 03:41:55.365667 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.365652 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 03:41:55.435356 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.435333 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:41:55.443382 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.443360 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:55.501290 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.501266 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 03:41:55.502818 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.502805 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:41:55.557007 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.556985 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 03:41:55.618512 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.618435 2579 apiserver.go:52] "Watching apiserver" Nov 01 03:41:55.642605 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.642575 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 03:41:55.653904 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.653883 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:41:55.747001 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.746974 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 03:41:55.747300 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.747283 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal","openshift-multus/multus-5g5xz","openshift-cluster-node-tuning-operator/tuned-x2ld9","openshift-dns/node-resolver-x7gtf","openshift-image-registry/node-ca-f79cb","openshift-network-diagnostics/network-check-target-r4j7n","openshift-ovn-kubernetes/ovnkube-node-8kr8k","kube-system/global-pull-secret-syncer-nwbf4","kube-system/konnectivity-agent-k2s25","kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757","openshift-multus/multus-additional-cni-plugins-74gr2","openshift-multus/network-metrics-daemon-h7fxk","openshift-network-operator/iptables-alerter-l8dmf"] Nov 01 03:41:55.749675 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.749653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.752746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.752726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.753991 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.753973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.755123 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.755105 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.756304 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.756287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.757561 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.757541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:55.757659 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.757609 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:41:55.758830 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.758816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.760269 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.760254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:55.760346 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.760304 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:41:55.761845 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.761825 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:55.763587 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.763570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.765673 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.765653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.767553 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.767534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.767645 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.767598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:41:55.790534 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.790486 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"5cc61107e38660d4c05b158edffa7de6","Type":"ContainerStarted","Data":"d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a"} Nov 01 03:41:55.791599 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.791578 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"63ad88aaffc4eb2dd4273e995e15f57f","Type":"ContainerStarted","Data":"249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5"} Nov 01 03:41:55.798540 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.798521 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 03:41:55.798626 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.798568 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 03:41:55.798626 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.798572 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 03:41:55.798703 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.798683 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h4w6m\"" Nov 01 03:41:55.844114 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.844080 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 03:41:55.858539 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:55.858667 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-netns\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.858667 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-registration-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.858667 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-kubelet\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.858786 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.858786 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.858786 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-socket-dir-parent\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.858786 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-systemd-units\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.858931 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-log-socket\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.858931 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38b835f0-2c16-4cee-8787-9914b169aff0-ovn-node-metrics-cert\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.858931 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkg2\" (UniqueName: \"kubernetes.io/projected/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kube-api-access-xbkg2\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.858931 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysconfig\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.858931 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.858931 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-dbus\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c23f988-ab24-48b8-b91d-c56a1d852f5d-hosts-file\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.858986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-sys-fs\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cnibin\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-kubelet-config\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-kubernetes\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-var-lib-kubelet\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.859180 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-tuned\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwb8h\" (UniqueName: \"kubernetes.io/projected/78bd8e89-b28f-4ce4-abaa-dc668b48014b-kube-api-access-fwb8h\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-multus\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-etc-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23f988-ab24-48b8-b91d-c56a1d852f5d-tmp-dir\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-systemd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-var-lib-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-tmp\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-ovn\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859472 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-netd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-env-overrides\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-conf-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-run\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b404292b-90b0-42b0-bc52-1fb2b5884250-serviceca\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4vll\" (UniqueName: \"kubernetes.io/projected/3b359175-b607-449b-94ee-86cd7923be08-kube-api-access-l4vll\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5r5\" (UniqueName: \"kubernetes.io/projected/38b835f0-2c16-4cee-8787-9914b169aff0-kube-api-access-mm5r5\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-socket-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-cni-binary-copy\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-netns\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.859836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7cvv\" (UniqueName: \"kubernetes.io/projected/8c23f988-ab24-48b8-b91d-c56a1d852f5d-kube-api-access-f7cvv\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-kubelet\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-script-lib\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.859980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q84ml\" (UniqueName: \"kubernetes.io/projected/915c8081-3df3-4ada-adca-74cb14da66de-kube-api-access-q84ml\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrmk\" (UniqueName: \"kubernetes.io/projected/67e6497d-1fe5-4ad9-89a5-18997c1bc527-kube-api-access-fbrmk\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b404292b-90b0-42b0-bc52-1fb2b5884250-host\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-multus-certs\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-sys\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-lib-modules\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-host\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b359175-b607-449b-94ee-86cd7923be08-host-slash\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-device-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-system-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-k8s-cni-cncf-io\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-node-log\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-bin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-hostroot\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-konnectivity-ca\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-cnibin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-etc-kubernetes\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-modprobe-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-conf\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b359175-b607-449b-94ee-86cd7923be08-iptables-alerter-script\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xrl\" (UniqueName: \"kubernetes.io/projected/b404292b-90b0-42b0-bc52-1fb2b5884250-kube-api-access-x4xrl\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-slash\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-systemd\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-agent-certs\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-system-cni-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-bin\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.860873 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-config\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.861348 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-multus-daemon-config\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.861348 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gsj\" (UniqueName: \"kubernetes.io/projected/df6cbdc5-6506-47c1-a44e-6d185d3fc716-kube-api-access-j4gsj\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:55.861348 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.860734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-binary-copy\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.896503 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.896430 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 03:41:55.896628 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.896504 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r2qm4" Nov 01 03:41:55.899167 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.899145 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 03:41:55.899167 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.899149 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 03:41:55.899530 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.899515 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v2zmn\"" Nov 01 03:41:55.899641 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.899630 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 03:41:55.905910 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.905894 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jjmfj\"" Nov 01 03:41:55.906016 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.905996 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 03:41:55.906140 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.906017 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 03:41:55.906140 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.906026 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-prgzf\"" Nov 01 03:41:55.906140 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.906062 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 03:41:55.906140 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.906067 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 03:41:55.906140 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.906073 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 03:41:55.906140 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.906114 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 03:41:55.945225 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945201 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 03:41:55.945314 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945244 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 03:41:55.945314 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945264 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 03:41:55.945373 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945314 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tmt47\"" Nov 01 03:41:55.945681 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945667 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 03:41:55.945721 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945682 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 03:41:55.945721 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945697 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 03:41:55.945800 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945726 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 03:41:55.945938 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945900 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 03:41:55.945938 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945924 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 03:41:55.945938 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945926 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 03:41:55.946128 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945947 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 03:41:55.946128 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.945968 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qv2jj\"" Nov 01 03:41:55.946128 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.946006 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 03:41:55.946128 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.946044 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dc6ls\"" Nov 01 03:41:55.946128 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.946104 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 03:41:55.952654 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.952628 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 03:41:55.952750 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.952699 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7tm86\"" Nov 01 03:41:55.952750 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.952729 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zn4h5\"" Nov 01 03:41:55.952833 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.952783 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 03:41:55.961478 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-k8s-cni-cncf-io\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961587 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-node-log\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.961587 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-bin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961587 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-hostroot\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961587 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-konnectivity-ca\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-k8s-cni-cncf-io\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961611 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-bin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-cnibin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-etc-kubernetes\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961617 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-hostroot\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961668 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-node-log\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-modprobe-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-cnibin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-conf\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961715 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-etc-kubernetes\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b359175-b607-449b-94ee-86cd7923be08-iptables-alerter-script\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xrl\" (UniqueName: \"kubernetes.io/projected/b404292b-90b0-42b0-bc52-1fb2b5884250-kube-api-access-x4xrl\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.961769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-slash\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-systemd\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-agent-certs\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-modprobe-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-system-cni-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961821 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-conf\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-bin\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-config\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-systemd\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-multus-daemon-config\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961886 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gsj\" (UniqueName: \"kubernetes.io/projected/df6cbdc5-6506-47c1-a44e-6d185d3fc716-kube-api-access-j4gsj\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-binary-copy\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.961977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-netns\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-registration-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-netns\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.962339 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-kubelet\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962106 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-system-cni-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962166 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-bin\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-slash\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-registration-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962285 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-kubelet\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-socket-dir-parent\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-systemd-units\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-log-socket\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38b835f0-2c16-4cee-8787-9914b169aff0-ovn-node-metrics-cert\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkg2\" (UniqueName: \"kubernetes.io/projected/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kube-api-access-xbkg2\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysconfig\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-systemd-units\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-socket-dir-parent\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.963054 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962460 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-dbus\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962497 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-log-socket\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962497 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysconfig\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c23f988-ab24-48b8-b91d-c56a1d852f5d-hosts-file\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-sys-fs\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962573 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c23f988-ab24-48b8-b91d-c56a1d852f5d-hosts-file\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cnibin\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-kubelet-config\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-sys-fs\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962536 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-dbus\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-kubernetes\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.962730 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-var-lib-kubelet\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.963746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962831 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-kubelet-config\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.962840 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs podName:df6cbdc5-6506-47c1-a44e-6d185d3fc716 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:56.462802062 +0000 UTC m=+3.234111279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs") pod "network-metrics-daemon-h7fxk" (UID: "df6cbdc5-6506-47c1-a44e-6d185d3fc716") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962887 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-kubernetes\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-konnectivity-ca\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-var-lib-kubelet\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962960 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.962983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-tuned\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwb8h\" (UniqueName: \"kubernetes.io/projected/78bd8e89-b28f-4ce4-abaa-dc668b48014b-kube-api-access-fwb8h\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cnibin\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-multus\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-etc-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23f988-ab24-48b8-b91d-c56a1d852f5d-tmp-dir\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-multus\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-systemd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-var-lib-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.964683 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-var-lib-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-binary-copy\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963278 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963312 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-multus-daemon-config\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-tmp\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-etc-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-systemd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-ovn\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-netd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-ovn\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963587 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23f988-ab24-48b8-b91d-c56a1d852f5d-tmp-dir\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-env-overrides\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-conf-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.963775 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-run\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.965458 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-netd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:55.963899 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret podName:128c7b38-4784-403c-be55-e2f1561c2f72 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:56.463881257 +0000 UTC m=+3.235190538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret") pod "global-pull-secret-syncer-nwbf4" (UID: "128c7b38-4784-403c-be55-e2f1561c2f72") : object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b404292b-90b0-42b0-bc52-1fb2b5884250-serviceca\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963938 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.963961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4vll\" (UniqueName: \"kubernetes.io/projected/3b359175-b607-449b-94ee-86cd7923be08-kube-api-access-l4vll\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5r5\" (UniqueName: \"kubernetes.io/projected/38b835f0-2c16-4cee-8787-9914b169aff0-kube-api-access-mm5r5\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-socket-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b359175-b607-449b-94ee-86cd7923be08-iptables-alerter-script\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-cni-binary-copy\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-netns\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964257 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-conf-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-config\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-netns\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966258 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964422 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-socket-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964471 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-env-overrides\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964489 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-run\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cvv\" (UniqueName: \"kubernetes.io/projected/8c23f988-ab24-48b8-b91d-c56a1d852f5d-kube-api-access-f7cvv\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-kubelet\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-script-lib\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q84ml\" (UniqueName: \"kubernetes.io/projected/915c8081-3df3-4ada-adca-74cb14da66de-kube-api-access-q84ml\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-kubelet\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964679 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrmk\" (UniqueName: \"kubernetes.io/projected/67e6497d-1fe5-4ad9-89a5-18997c1bc527-kube-api-access-fbrmk\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b404292b-90b0-42b0-bc52-1fb2b5884250-host\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964782 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-cni-binary-copy\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-multus-certs\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-sys\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964834 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b404292b-90b0-42b0-bc52-1fb2b5884250-host\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-lib-modules\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964888 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-multus-certs\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964896 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-sys\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.966890 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.964920 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965019 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-lib-modules\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b404292b-90b0-42b0-bc52-1fb2b5884250-serviceca\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-host\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-host\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b359175-b607-449b-94ee-86cd7923be08-host-slash\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-device-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-system-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b359175-b607-449b-94ee-86cd7923be08-host-slash\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965254 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-device-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965286 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-system-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.965619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-script-lib\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.966747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-tuned\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.967062 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-agent-certs\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.967152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-tmp\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:55.967467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:55.967178 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38b835f0-2c16-4cee-8787-9914b169aff0-ovn-node-metrics-cert\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:56.018230 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.018197 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 03:41:56.018230 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.018229 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 03:41:56.018416 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.018243 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vqk9g for pod openshift-network-diagnostics/network-check-target-r4j7n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:56.018469 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.018433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g podName:8d064689-ea69-4221-b860-9242ee3f80a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:56.518410407 +0000 UTC m=+3.289719623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vqk9g" (UniqueName: "kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g") pod "network-check-target-r4j7n" (UID: "8d064689-ea69-4221-b860-9242ee3f80a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:56.020680 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.020657 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwb8h\" (UniqueName: \"kubernetes.io/projected/78bd8e89-b28f-4ce4-abaa-dc668b48014b-kube-api-access-fwb8h\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:56.020760 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.020698 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xrl\" (UniqueName: \"kubernetes.io/projected/b404292b-90b0-42b0-bc52-1fb2b5884250-kube-api-access-x4xrl\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:56.020760 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.020664 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkg2\" (UniqueName: \"kubernetes.io/projected/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kube-api-access-xbkg2\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:56.020841 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.020780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gsj\" (UniqueName: \"kubernetes.io/projected/df6cbdc5-6506-47c1-a44e-6d185d3fc716-kube-api-access-j4gsj\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:56.024096 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.024069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q84ml\" (UniqueName: \"kubernetes.io/projected/915c8081-3df3-4ada-adca-74cb14da66de-kube-api-access-q84ml\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:41:56.024360 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.024223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4vll\" (UniqueName: \"kubernetes.io/projected/3b359175-b607-449b-94ee-86cd7923be08-kube-api-access-l4vll\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:56.024360 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.024283 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5r5\" (UniqueName: \"kubernetes.io/projected/38b835f0-2c16-4cee-8787-9914b169aff0-kube-api-access-mm5r5\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:56.025640 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.025614 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrmk\" (UniqueName: \"kubernetes.io/projected/67e6497d-1fe5-4ad9-89a5-18997c1bc527-kube-api-access-fbrmk\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:56.025787 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.025768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cvv\" (UniqueName: \"kubernetes.io/projected/8c23f988-ab24-48b8-b91d-c56a1d852f5d-kube-api-access-f7cvv\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:56.067122 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.067094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:41:56.069223 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.069206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:41:56.074498 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.074483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5g5xz" Nov 01 03:41:56.079365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.079346 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:41:56.084891 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.084875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:41:56.091147 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.091122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:41:56.093529 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:56.093507 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e6497d_1fe5_4ad9_89a5_18997c1bc527.slice/crio-55f355756075d7c7c218b6323dc7f73327f7b4a4e711fb5f87d751eb25dc95a3 WatchSource:0}: Error finding container 55f355756075d7c7c218b6323dc7f73327f7b4a4e711fb5f87d751eb25dc95a3: Status 404 returned error can't find the container with id 55f355756075d7c7c218b6323dc7f73327f7b4a4e711fb5f87d751eb25dc95a3 Nov 01 03:41:56.098652 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.098626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:41:56.101253 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:56.101232 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b359175_b607_449b_94ee_86cd7923be08.slice/crio-b835c3d5f549f025d891935df7006d3210d293121f7deedc4be6ea290a75b078 WatchSource:0}: Error finding container b835c3d5f549f025d891935df7006d3210d293121f7deedc4be6ea290a75b078: Status 404 returned error can't find the container with id b835c3d5f549f025d891935df7006d3210d293121f7deedc4be6ea290a75b078 Nov 01 03:41:56.107438 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:56.107415 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bed51d_4e8a_4f9c_8c9b_d0aaf0691216.slice/crio-b6ebd357cd9ac717b5fac0fcc1dc30fdfefac89ef3c9311a804e41c055e4b6e8 WatchSource:0}: Error finding container b6ebd357cd9ac717b5fac0fcc1dc30fdfefac89ef3c9311a804e41c055e4b6e8: Status 404 returned error can't find the container with id b6ebd357cd9ac717b5fac0fcc1dc30fdfefac89ef3c9311a804e41c055e4b6e8 Nov 01 03:41:56.121048 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.121012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:41:56.129091 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.129069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:41:56.130635 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:56.130606 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bd8e89_b28f_4ce4_abaa_dc668b48014b.slice/crio-b22032917b559a73c2704d72d879eb3fd30b66666faee15df6737a4515baecd4 WatchSource:0}: Error finding container b22032917b559a73c2704d72d879eb3fd30b66666faee15df6737a4515baecd4: Status 404 returned error can't find the container with id b22032917b559a73c2704d72d879eb3fd30b66666faee15df6737a4515baecd4 Nov 01 03:41:56.136928 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:41:56.136908 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b835f0_2c16_4cee_8787_9914b169aff0.slice/crio-4efc45b23dedf56d295be7a90bd7459d2ba8a9a5178117bfe43f127612e0e9c8 WatchSource:0}: Error finding container 4efc45b23dedf56d295be7a90bd7459d2ba8a9a5178117bfe43f127612e0e9c8: Status 404 returned error can't find the container with id 4efc45b23dedf56d295be7a90bd7459d2ba8a9a5178117bfe43f127612e0e9c8 Nov 01 03:41:56.468948 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.468913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:56.469151 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.468962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:56.469151 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.469095 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:56.469151 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.469111 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:56.469151 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.469150 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret podName:128c7b38-4784-403c-be55-e2f1561c2f72 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:57.469132987 +0000 UTC m=+4.240442205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret") pod "global-pull-secret-syncer-nwbf4" (UID: "128c7b38-4784-403c-be55-e2f1561c2f72") : object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:56.469328 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.469170 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs podName:df6cbdc5-6506-47c1-a44e-6d185d3fc716 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:57.469156487 +0000 UTC m=+4.240465711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs") pod "network-metrics-daemon-h7fxk" (UID: "df6cbdc5-6506-47c1-a44e-6d185d3fc716") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:56.570252 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.570221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:56.570413 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.570378 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 03:41:56.570413 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.570399 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 03:41:56.570413 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.570409 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vqk9g for pod openshift-network-diagnostics/network-check-target-r4j7n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:56.570505 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:56.570460 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g podName:8d064689-ea69-4221-b860-9242ee3f80a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:57.570446173 +0000 UTC m=+4.341755396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vqk9g" (UniqueName: "kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g") pod "network-check-target-r4j7n" (UID: "8d064689-ea69-4221-b860-9242ee3f80a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:56.795579 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.795481 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2s25" event={"ID":"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216","Type":"ContainerStarted","Data":"b6ebd357cd9ac717b5fac0fcc1dc30fdfefac89ef3c9311a804e41c055e4b6e8"} Nov 01 03:41:56.798199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.798171 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x7gtf" event={"ID":"8c23f988-ab24-48b8-b91d-c56a1d852f5d","Type":"ContainerStarted","Data":"37fe08941fdd657eecd19192333b5b2988fdb45578c4f864cd87dc1a0e9131cd"} Nov 01 03:41:56.800023 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.799996 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"4efc45b23dedf56d295be7a90bd7459d2ba8a9a5178117bfe43f127612e0e9c8"} Nov 01 03:41:56.801490 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.801466 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerStarted","Data":"b22032917b559a73c2704d72d879eb3fd30b66666faee15df6737a4515baecd4"} Nov 01 03:41:56.802786 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.802725 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l8dmf" event={"ID":"3b359175-b607-449b-94ee-86cd7923be08","Type":"ContainerStarted","Data":"b835c3d5f549f025d891935df7006d3210d293121f7deedc4be6ea290a75b078"} Nov 01 03:41:56.804321 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.804295 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f79cb" event={"ID":"b404292b-90b0-42b0-bc52-1fb2b5884250","Type":"ContainerStarted","Data":"a74c36ea1e9b0a2b54956e883fc150c8225329ecd15713de64ac73848a3dcb06"} Nov 01 03:41:56.806307 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.806281 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" event={"ID":"67e6497d-1fe5-4ad9-89a5-18997c1bc527","Type":"ContainerStarted","Data":"55f355756075d7c7c218b6323dc7f73327f7b4a4e711fb5f87d751eb25dc95a3"} Nov 01 03:41:56.807750 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.807722 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5g5xz" event={"ID":"915c8081-3df3-4ada-adca-74cb14da66de","Type":"ContainerStarted","Data":"0074608fcbb04640a9f12e496d37ecf08fe99646cce29b5ee6dc19098bcef04a"} Nov 01 03:41:56.808996 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.808977 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" event={"ID":"be542312-aee2-46b1-aa3d-f1c5e07cd59c","Type":"ContainerStarted","Data":"1805efbbea12a1ecdaa692cc89eca2527e953f2ec24b3ca720aa26f5a83e1d44"} Nov 01 03:41:56.899846 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.899766 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:36:55 +0000 UTC" deadline="2028-03-28 19:42:23.399453849 +0000 UTC" Nov 01 03:41:56.899846 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:56.899841 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12328h0m26.499616243s" Nov 01 03:41:57.477663 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.477133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:57.477663 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.477248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:57.477663 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.477396 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:57.477663 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.477533 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs podName:df6cbdc5-6506-47c1-a44e-6d185d3fc716 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:59.477492173 +0000 UTC m=+6.248801386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs") pod "network-metrics-daemon-h7fxk" (UID: "df6cbdc5-6506-47c1-a44e-6d185d3fc716") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:57.477663 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.477601 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:57.477663 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.477637 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret podName:128c7b38-4784-403c-be55-e2f1561c2f72 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:59.477625423 +0000 UTC m=+6.248934624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret") pod "global-pull-secret-syncer-nwbf4" (UID: "128c7b38-4784-403c-be55-e2f1561c2f72") : object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:57.577994 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.577941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:57.578240 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.578170 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 03:41:57.578240 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.578191 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 03:41:57.578240 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.578204 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vqk9g for pod openshift-network-diagnostics/network-check-target-r4j7n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:57.578471 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.578327 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g podName:8d064689-ea69-4221-b860-9242ee3f80a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:41:59.578245125 +0000 UTC m=+6.349554354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vqk9g" (UniqueName: "kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g") pod "network-check-target-r4j7n" (UID: "8d064689-ea69-4221-b860-9242ee3f80a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:57.785204 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.785168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:57.785204 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.785200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:57.785446 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.785307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:41:57.785446 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.785405 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:41:57.785697 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.785582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:57.785697 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:57.785683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:41:57.900858 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.900786 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:36:55 +0000 UTC" deadline="2028-06-03 14:23:10.836232723 +0000 UTC" Nov 01 03:41:57.900858 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:57.900839 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13930h41m12.935397736s" Nov 01 03:41:59.492923 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.492720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:59.493486 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.492935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:59.493486 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.493084 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:59.493486 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.493149 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret podName:128c7b38-4784-403c-be55-e2f1561c2f72 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.493131094 +0000 UTC m=+10.264440308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret") pod "global-pull-secret-syncer-nwbf4" (UID: "128c7b38-4784-403c-be55-e2f1561c2f72") : object "kube-system"/"original-pull-secret" not registered Nov 01 03:41:59.493486 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.493217 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:59.493486 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.493250 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs podName:df6cbdc5-6506-47c1-a44e-6d185d3fc716 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.493238581 +0000 UTC m=+10.264547781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs") pod "network-metrics-daemon-h7fxk" (UID: "df6cbdc5-6506-47c1-a44e-6d185d3fc716") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:41:59.593631 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.593551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:59.593796 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.593772 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 03:41:59.593796 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.593791 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 03:41:59.593911 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.593805 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vqk9g for pod openshift-network-diagnostics/network-check-target-r4j7n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:59.593911 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.593869 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g podName:8d064689-ea69-4221-b860-9242ee3f80a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:03.593850025 +0000 UTC m=+10.365159230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vqk9g" (UniqueName: "kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g") pod "network-check-target-r4j7n" (UID: "8d064689-ea69-4221-b860-9242ee3f80a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:41:59.785241 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.785208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:41:59.785439 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.785357 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:41:59.785439 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.785416 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:41:59.785564 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.785484 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:41:59.785564 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.785525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:41:59.785665 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:41:59.785582 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:41:59.820016 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.819977 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"63ad88aaffc4eb2dd4273e995e15f57f","Type":"ContainerStarted","Data":"c0af13e51e36c5cd9c6e219c3a7998c37b8e160970209fb57e9e5e0832f72ddb"} Nov 01 03:41:59.823419 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.823381 2579 generic.go:356] "Generic (PLEG): container finished" podID="5cc61107e38660d4c05b158edffa7de6" containerID="9019fc5505e4a141bdec96f3c296a2343311f06e0b68bb3b171948c9be2fb79b" exitCode=0 Nov 01 03:41:59.823545 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.823475 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"5cc61107e38660d4c05b158edffa7de6","Type":"ContainerDied","Data":"9019fc5505e4a141bdec96f3c296a2343311f06e0b68bb3b171948c9be2fb79b"} Nov 01 03:41:59.830066 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.830021 2579 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="ab5438851fbf8abaa007558c6df12d83cd594ea5c41d5faf84927d0c1a804ab5" exitCode=0 Nov 01 03:41:59.830173 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.830087 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerDied","Data":"ab5438851fbf8abaa007558c6df12d83cd594ea5c41d5faf84927d0c1a804ab5"} Nov 01 03:41:59.927365 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:41:59.927305 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" podStartSLOduration=4.927287504 podStartE2EDuration="4.927287504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:59.926269833 +0000 UTC m=+6.697579061" watchObservedRunningTime="2026-11-01 03:41:59.927287504 +0000 UTC m=+6.698596728" Nov 01 03:42:00.836065 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:00.836008 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2s25" event={"ID":"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216","Type":"ContainerStarted","Data":"95497192e8049635f998cf2345d9b375cec90e855e1831fb9835e4bf8ae951e6"} Nov 01 03:42:00.839465 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:00.839427 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"5cc61107e38660d4c05b158edffa7de6","Type":"ContainerStarted","Data":"d8a04d442de6eec38b26e045ba38c2cd2dedac85b2e152395c16dd1b83da0a6d"} Nov 01 03:42:00.844734 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:00.844711 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f79cb" event={"ID":"b404292b-90b0-42b0-bc52-1fb2b5884250","Type":"ContainerStarted","Data":"a440b8a7ee65f4dc3175aeac8e8d48c13c69bdf489b225a10be66864aab3aeb1"} Nov 01 03:42:00.945251 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:00.945139 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k2s25" podStartSLOduration=3.898747587 podStartE2EDuration="7.945118935s" totalImagesPullingTime="4.046371348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:00.944368698 +0000 UTC m=+7.715677925" watchObservedRunningTime="2026-11-01 03:42:00.945118935 +0000 UTC m=+7.716428161" Nov 01 03:42:01.053588 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.053524 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" podStartSLOduration=6.053505851 podStartE2EDuration="6.053505851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:01.052886381 +0000 UTC m=+7.824195604" watchObservedRunningTime="2026-11-01 03:42:01.053505851 +0000 UTC m=+7.824815073" Nov 01 03:42:01.159277 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.159219 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f79cb" podStartSLOduration=3.6058862879999998 podStartE2EDuration="8.159202003s" totalImagesPullingTime="4.553315715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:01.15913157 +0000 UTC m=+7.930440793" watchObservedRunningTime="2026-11-01 03:42:01.159202003 +0000 UTC m=+7.930511228" Nov 01 03:42:01.527302 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.527213 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:42:01.528433 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.528405 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:42:01.787172 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.787075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:01.787334 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:01.787207 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:01.787587 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.787564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:01.787683 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:01.787657 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:01.787738 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.787720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:01.787895 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:01.787802 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:01.849836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:01.848908 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" event={"ID":"be542312-aee2-46b1-aa3d-f1c5e07cd59c","Type":"ContainerStarted","Data":"3edfb68efb7803e2c0d30ab1d0a7ac5ad8c00572977fa50b870b3573eda174a1"} Nov 01 03:42:02.851248 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:02.851217 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 03:42:03.528089 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.528054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:03.528276 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.528134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:03.528276 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.528238 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:42:03.528276 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.528242 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 03:42:03.528441 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.528290 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs podName:df6cbdc5-6506-47c1-a44e-6d185d3fc716 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:11.528276961 +0000 UTC m=+18.299586161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs") pod "network-metrics-daemon-h7fxk" (UID: "df6cbdc5-6506-47c1-a44e-6d185d3fc716") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:42:03.528441 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.528307 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret podName:128c7b38-4784-403c-be55-e2f1561c2f72 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:11.52829991 +0000 UTC m=+18.299609110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret") pod "global-pull-secret-syncer-nwbf4" (UID: "128c7b38-4784-403c-be55-e2f1561c2f72") : object "kube-system"/"original-pull-secret" not registered Nov 01 03:42:03.595402 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.595354 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 03:42:03.629397 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.629290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:03.629521 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.629486 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 03:42:03.629521 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.629509 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 03:42:03.629697 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.629522 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vqk9g for pod openshift-network-diagnostics/network-check-target-r4j7n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:42:03.629697 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.629609 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g podName:8d064689-ea69-4221-b860-9242ee3f80a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:11.62959 +0000 UTC m=+18.400899200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vqk9g" (UniqueName: "kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g") pod "network-check-target-r4j7n" (UID: "8d064689-ea69-4221-b860-9242ee3f80a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:42:03.723815 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.723708 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:42:03.595374543Z","UUID":"04c3e2b8-f384-4d50-9c54-32839201fc28","Handler":null,"Name":"","Endpoint":""} Nov 01 03:42:03.729465 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.729438 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 03:42:03.729465 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.729470 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 03:42:03.787206 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.787177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:03.787392 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.787302 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:03.787392 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.787371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:03.787522 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.787436 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:03.787522 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.787506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:03.787627 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:03.787589 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:03.855831 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.855796 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" event={"ID":"be542312-aee2-46b1-aa3d-f1c5e07cd59c","Type":"ContainerStarted","Data":"719cb9439d135b8f5fa52ef943d167b215e0a9d791fae67e011291d20f7e06c2"} Nov 01 03:42:03.858348 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.858320 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x7gtf" event={"ID":"8c23f988-ab24-48b8-b91d-c56a1d852f5d","Type":"ContainerStarted","Data":"208dbd8f4202d543bacb45ab5a16555d1e070526bc0d2eadfb0a68ae4ed230c1"} Nov 01 03:42:03.966744 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:03.966687 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x7gtf" podStartSLOduration=4.112400829 podStartE2EDuration="10.966669503s" totalImagesPullingTime="6.854268674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:03.966101025 +0000 UTC m=+10.737410249" watchObservedRunningTime="2026-11-01 03:42:03.966669503 +0000 UTC m=+10.737978726" Nov 01 03:42:04.862958 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:04.862699 2579 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="917264426d5c7b7c9f178daf09f643ece124f507b75e971b96124360246f185c" exitCode=0 Nov 01 03:42:04.862958 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:04.862752 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerDied","Data":"917264426d5c7b7c9f178daf09f643ece124f507b75e971b96124360246f185c"} Nov 01 03:42:04.864951 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:04.864743 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l8dmf" event={"ID":"3b359175-b607-449b-94ee-86cd7923be08","Type":"ContainerStarted","Data":"5a42343b966ccc2f224ddf0162cc18b4effb3614b5730d947bfcba7e025df073"} Nov 01 03:42:05.087931 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:05.087876 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l8dmf" podStartSLOduration=5.41016292 podStartE2EDuration="12.087860894s" totalImagesPullingTime="6.677697974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:05.087393679 +0000 UTC m=+11.858702900" watchObservedRunningTime="2026-11-01 03:42:05.087860894 +0000 UTC m=+11.859170115" Nov 01 03:42:05.785532 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:05.785494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:05.785735 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:05.785572 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:05.785735 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:05.785581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:05.785735 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:05.785686 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:05.785896 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:05.785775 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:05.785896 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:05.785859 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:05.869748 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:05.869710 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" event={"ID":"be542312-aee2-46b1-aa3d-f1c5e07cd59c","Type":"ContainerStarted","Data":"109549de684a03766384b30e9533b1c27f56a5ecaa24723f0327ec4a96bbefd0"} Nov 01 03:42:05.981326 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:05.981273 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" podStartSLOduration=4.028629765 podStartE2EDuration="12.981260082s" totalImagesPullingTime="8.952630317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:05.980950651 +0000 UTC m=+12.752259876" watchObservedRunningTime="2026-11-01 03:42:05.981260082 +0000 UTC m=+12.752569305" Nov 01 03:42:06.874161 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:06.874121 2579 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="02047441698c77fc6a5614577257b090bfebdcd58b1c9461372f7777924a309a" exitCode=0 Nov 01 03:42:06.874776 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:06.874748 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerDied","Data":"02047441698c77fc6a5614577257b090bfebdcd58b1c9461372f7777924a309a"} Nov 01 03:42:07.785075 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:07.784781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:07.785075 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:07.784826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:07.785287 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:07.785090 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:07.785287 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:07.784863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:07.785287 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:07.785189 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:07.785287 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:07.785249 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:07.878508 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:07.878466 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" event={"ID":"67e6497d-1fe5-4ad9-89a5-18997c1bc527","Type":"ContainerStarted","Data":"d7ffb9c14e91dcc306d3a3ca870f8915611578729d5471642254f6358922b2f6"} Nov 01 03:42:07.990329 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:07.990265 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" podStartSLOduration=3.614154642 podStartE2EDuration="14.990246472s" totalImagesPullingTime="11.37609183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:07.990030582 +0000 UTC m=+14.761339805" watchObservedRunningTime="2026-11-01 03:42:07.990246472 +0000 UTC m=+14.761555738" Nov 01 03:42:08.883792 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:08.883752 2579 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="f5188af0ca035aa1c209e861fda18cf9bebf1c7ef9f675d2ec51e3fe1b160e72" exitCode=0 Nov 01 03:42:08.884274 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:08.883837 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerDied","Data":"f5188af0ca035aa1c209e861fda18cf9bebf1c7ef9f675d2ec51e3fe1b160e72"} Nov 01 03:42:09.784928 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:09.784893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:09.785139 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:09.784893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:09.785139 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:09.785017 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:09.785139 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:09.785055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:09.785256 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:09.785157 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:09.785293 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:09.785255 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:09.887426 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:09.887387 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5g5xz" event={"ID":"915c8081-3df3-4ada-adca-74cb14da66de","Type":"ContainerStarted","Data":"9d063fc410e25dce2fd3bcd7fab4ef7048782df756d5cdd25fd4880c9b3614c6"} Nov 01 03:42:09.998749 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:09.998697 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5g5xz" podStartSLOduration=3.917906255 podStartE2EDuration="16.998683025s" totalImagesPullingTime="13.08077677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:09.998538069 +0000 UTC m=+16.769847292" watchObservedRunningTime="2026-11-01 03:42:09.998683025 +0000 UTC m=+16.769992247" Nov 01 03:42:11.587832 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:11.587786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:11.588335 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:11.587855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:11.588335 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.587928 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:42:11.588335 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.587956 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 03:42:11.588335 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.588006 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs podName:df6cbdc5-6506-47c1-a44e-6d185d3fc716 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:27.587983412 +0000 UTC m=+34.359292625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs") pod "network-metrics-daemon-h7fxk" (UID: "df6cbdc5-6506-47c1-a44e-6d185d3fc716") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:42:11.588335 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.588027 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret podName:128c7b38-4784-403c-be55-e2f1561c2f72 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:27.58801675 +0000 UTC m=+34.359325954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret") pod "global-pull-secret-syncer-nwbf4" (UID: "128c7b38-4784-403c-be55-e2f1561c2f72") : object "kube-system"/"original-pull-secret" not registered Nov 01 03:42:11.688274 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:11.688234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:11.688436 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.688417 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 03:42:11.688484 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.688440 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 03:42:11.688484 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.688455 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vqk9g for pod openshift-network-diagnostics/network-check-target-r4j7n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:42:11.688588 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.688521 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g podName:8d064689-ea69-4221-b860-9242ee3f80a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:27.688503174 +0000 UTC m=+34.459812388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vqk9g" (UniqueName: "kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g") pod "network-check-target-r4j7n" (UID: "8d064689-ea69-4221-b860-9242ee3f80a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:42:11.785122 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:11.785086 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:11.785122 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:11.785111 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:11.785326 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.785294 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:11.785635 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:11.785615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:11.785760 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.785738 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:11.785836 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:11.785817 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:12.897637 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.897467 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:42:12.898137 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.897948 2579 generic.go:356] "Generic (PLEG): container finished" podID="38b835f0-2c16-4cee-8787-9914b169aff0" containerID="90be2596562c5226bdfa857f98a60a6a72998c1269b3312d16169e64897c5893" exitCode=1 Nov 01 03:42:12.898137 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.898018 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"7e5bff1438076edd36f35b1433aca5d96ad1dd78a4ed5bcfc0d88ba2a91b8d36"} Nov 01 03:42:12.898137 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.898101 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"cda38d919641a14d25d191997b60297654c28ca982d4b0fe36ed106f837fadbb"} Nov 01 03:42:12.898266 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.898147 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"aecb1d2ecfc1042f11af0ff0662ee315baa0dd016a3ce56c21f495caf5c8aa3e"} Nov 01 03:42:12.898266 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.898189 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"14ba3e8fdd531d88cd0dea5859fee4ec11cc2e1d00f8a2607c1c107ceed940d7"} Nov 01 03:42:12.898266 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.898222 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerDied","Data":"90be2596562c5226bdfa857f98a60a6a72998c1269b3312d16169e64897c5893"} Nov 01 03:42:12.898363 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.898272 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"b58ced0f97618464055f2a1b7074f59b890dd4535426044b0f9e140e5810bac8"} Nov 01 03:42:12.900802 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.900781 2579 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="facc551ac70f0bd072c7ba8cb023a96d1e434e0d5c8630ff0a5dec92d44cc1f2" exitCode=0 Nov 01 03:42:12.900891 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:12.900825 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerDied","Data":"facc551ac70f0bd072c7ba8cb023a96d1e434e0d5c8630ff0a5dec92d44cc1f2"} Nov 01 03:42:13.785739 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:13.785700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:13.785879 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:13.785829 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:13.785915 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:13.785881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:13.785943 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:13.785923 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:13.786029 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:13.786010 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:13.786130 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:13.786112 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:13.905447 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:13.905418 2579 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="6dfcf848db2362c3ce7be8ad965079428218e6da8ccee56a75a4f1f87f31fcb9" exitCode=0 Nov 01 03:42:13.905880 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:13.905462 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerDied","Data":"6dfcf848db2362c3ce7be8ad965079428218e6da8ccee56a75a4f1f87f31fcb9"} Nov 01 03:42:14.685631 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:14.685598 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:42:14.685783 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:14.685731 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 03:42:14.686213 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:14.686189 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:42:14.910371 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:14.910344 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:42:14.910870 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:14.910696 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"0090d311eef03657a13c0b623953e624d2d50aba5878228cf2a5bb495e8d4a4b"} Nov 01 03:42:14.913906 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:14.913881 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-74gr2" event={"ID":"78bd8e89-b28f-4ce4-abaa-dc668b48014b","Type":"ContainerStarted","Data":"96e9324370583c67d42335eeaa081e1d42e26ed051ce4c23361aa74d9d45bc7c"} Nov 01 03:42:15.031237 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:15.031184 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-74gr2" podStartSLOduration=8.453405297 podStartE2EDuration="22.031170096s" totalImagesPullingTime="13.577764799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:15.030651193 +0000 UTC m=+21.801960428" watchObservedRunningTime="2026-11-01 03:42:15.031170096 +0000 UTC m=+21.802479317" Nov 01 03:42:15.784580 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:15.784548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:15.784755 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:15.784582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:15.784755 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:15.784651 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:15.784755 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:15.784684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:15.784883 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:15.784782 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:15.784913 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:15.784891 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:16.922274 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:16.922250 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:42:17.785118 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.784875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:17.785265 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.784927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:17.785265 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:17.785202 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:17.785265 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.784956 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:17.785424 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:17.785253 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:17.785424 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:17.785402 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:17.928193 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.928171 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:42:17.928621 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.928531 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"02192a52f92e031b9766f2f404aa3c9640a7fdfa0e89f61d1aae16e24154e76c"} Nov 01 03:42:17.928829 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.928812 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:42:17.928927 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.928872 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:42:17.929002 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.928988 2579 scope.go:122] "RemoveContainer" containerID="90be2596562c5226bdfa857f98a60a6a72998c1269b3312d16169e64897c5893" Nov 01 03:42:17.946902 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:17.946884 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:42:18.934216 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:18.934188 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:42:18.934641 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:18.934566 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" event={"ID":"38b835f0-2c16-4cee-8787-9914b169aff0","Type":"ContainerStarted","Data":"b926cf53aedeb20873728667dc3885f0866a93dc9c518e455414a08b6fab1815"} Nov 01 03:42:18.934844 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:18.934827 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:42:18.949911 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:18.949887 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:42:19.102844 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:19.102794 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" podStartSLOduration=9.033942869 podStartE2EDuration="25.102781652s" totalImagesPullingTime="16.068838783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:19.102083988 +0000 UTC m=+25.873393211" watchObservedRunningTime="2026-11-01 03:42:19.102781652 +0000 UTC m=+25.874090873" Nov 01 03:42:19.784897 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:19.784862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:19.785107 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:19.784992 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:19.785107 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:19.785016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:19.785224 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:19.785129 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:19.785224 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:19.785152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:19.785224 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:19.785192 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:21.785359 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:21.785327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:21.785838 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:21.785362 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:21.785838 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:21.785422 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:21.785838 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:21.785484 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:21.785838 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:21.785540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:21.785838 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:21.785612 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:23.786199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:23.786168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:23.786619 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:23.786248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:23.786619 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:23.786280 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:23.786619 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:23.786311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:23.786619 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:23.786403 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:23.786619 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:23.786492 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:25.785024 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:25.784986 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:25.785475 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:25.785067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:25.785475 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:25.785150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:25.785475 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:25.785272 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:25.785475 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:25.785372 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:25.785475 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:25.785444 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:27.603435 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:27.603407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:27.603435 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:27.603441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:27.603853 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.603544 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 03:42:27.603853 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.603556 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:42:27.603853 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.603595 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret podName:128c7b38-4784-403c-be55-e2f1561c2f72 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:59.603582219 +0000 UTC m=+66.374891432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret") pod "global-pull-secret-syncer-nwbf4" (UID: "128c7b38-4784-403c-be55-e2f1561c2f72") : object "kube-system"/"original-pull-secret" not registered Nov 01 03:42:27.603853 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.603613 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs podName:df6cbdc5-6506-47c1-a44e-6d185d3fc716 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:59.603600952 +0000 UTC m=+66.374910166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs") pod "network-metrics-daemon-h7fxk" (UID: "df6cbdc5-6506-47c1-a44e-6d185d3fc716") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 03:42:27.704071 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:27.704020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:27.704235 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.704149 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 03:42:27.704235 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.704162 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 03:42:27.704235 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.704172 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vqk9g for pod openshift-network-diagnostics/network-check-target-r4j7n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:42:27.704235 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.704215 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g podName:8d064689-ea69-4221-b860-9242ee3f80a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:42:59.704202997 +0000 UTC m=+66.475512197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vqk9g" (UniqueName: "kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g") pod "network-check-target-r4j7n" (UID: "8d064689-ea69-4221-b860-9242ee3f80a0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 03:42:27.785332 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:27.785302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:27.785482 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:27.785307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:27.785482 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.785424 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:27.785482 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.785472 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:27.785584 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:27.785311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:27.785584 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:27.785539 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:28.585710 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:28.585682 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nwbf4"] Nov 01 03:42:28.585862 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:28.585783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:28.585912 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:28.585874 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:28.589153 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:28.589114 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r4j7n"] Nov 01 03:42:28.589402 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:28.589388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:28.589886 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:28.589863 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7fxk"] Nov 01 03:42:28.590360 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:28.590141 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:28.590360 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:28.590212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:28.590360 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:28.590308 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:30.396262 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:30.396070 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x7gtf_8c23f988-ab24-48b8-b91d-c56a1d852f5d/dns-node-resolver/0.log" Nov 01 03:42:30.428865 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:30.428840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x7gtf_8c23f988-ab24-48b8-b91d-c56a1d852f5d/dns-node-resolver/0.log" Nov 01 03:42:30.785352 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:30.785322 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:30.785352 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:30.785337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:30.785548 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:30.785325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:30.785548 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:30.785425 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:30.785548 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:30.785506 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:30.785644 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:30.785592 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:31.431631 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:31.431603 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f79cb_b404292b-90b0-42b0-bc52-1fb2b5884250/node-ca/0.log" Nov 01 03:42:31.579642 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:31.579611 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f79cb_b404292b-90b0-42b0-bc52-1fb2b5884250/node-ca/0.log" Nov 01 03:42:32.784748 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:32.784719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:32.785083 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:32.784719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:32.785083 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:32.784838 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h7fxk" podUID="df6cbdc5-6506-47c1-a44e-6d185d3fc716" Nov 01 03:42:32.785083 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:32.784718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:32.785083 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:32.784889 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4j7n" podUID="8d064689-ea69-4221-b860-9242ee3f80a0" Nov 01 03:42:32.785083 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:42:32.784975 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nwbf4" podUID="128c7b38-4784-403c-be55-e2f1561c2f72" Nov 01 03:42:32.999813 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:32.999781 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeReady" Nov 01 03:42:32.999964 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:32.999894 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 03:42:33.107770 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.107738 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lzs95"] Nov 01 03:42:33.110092 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.110068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.153837 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.153813 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fvvjf"] Nov 01 03:42:33.155503 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.155489 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.159374 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.159336 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 03:42:33.159374 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.159348 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 03:42:33.159374 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.159361 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8fn5p\"" Nov 01 03:42:33.159374 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.159343 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 03:42:33.159374 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.159364 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 03:42:33.167900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.167878 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fvvjf"] Nov 01 03:42:33.170750 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.170722 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzs95"] Nov 01 03:42:33.195726 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.195702 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7kp4z"] Nov 01 03:42:33.197524 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.197506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.240840 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.240812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-crio-socket\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.240970 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.240870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j95j2\" (UniqueName: \"kubernetes.io/projected/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-api-access-j95j2\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.240970 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.240888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-data-volume\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.240970 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.240956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54c97174-d65c-4516-923f-c504861b0196-cert\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.241105 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.241049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r48b\" (UniqueName: \"kubernetes.io/projected/54c97174-d65c-4516-923f-c504861b0196-kube-api-access-9r48b\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.241105 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.241088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.241161 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.241108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.255546 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.255495 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7kp4z"] Nov 01 03:42:33.259473 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.259451 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 03:42:33.259473 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.259450 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 03:42:33.259660 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.259478 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx8x8\"" Nov 01 03:42:33.259660 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.259486 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 03:42:33.259660 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.259589 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 03:42:33.259660 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.259597 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 03:42:33.259660 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.259630 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2mnqj\"" Nov 01 03:42:33.342005 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.341970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j95j2\" (UniqueName: \"kubernetes.io/projected/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-api-access-j95j2\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.342005 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-data-volume\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.342268 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-metrics-tls\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.342268 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54c97174-d65c-4516-923f-c504861b0196-cert\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.342268 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r48b\" (UniqueName: \"kubernetes.io/projected/54c97174-d65c-4516-923f-c504861b0196-kube-api-access-9r48b\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.342268 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.342467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.342467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-config-volume\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.342467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-crio-socket\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.342467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342408 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-data-volume\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.342467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-tmp-dir\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.342681 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56dwj\" (UniqueName: \"kubernetes.io/projected/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-kube-api-access-56dwj\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.342681 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342612 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-crio-socket\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.342926 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.342907 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.346220 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.346195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.346320 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.346280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54c97174-d65c-4516-923f-c504861b0196-cert\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.399767 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.399727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j95j2\" (UniqueName: \"kubernetes.io/projected/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-api-access-j95j2\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.399767 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.399750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r48b\" (UniqueName: \"kubernetes.io/projected/54c97174-d65c-4516-923f-c504861b0196-kube-api-access-9r48b\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.421618 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.421590 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:42:33.444216 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.443774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-config-volume\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.444216 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.443831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-tmp-dir\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.444216 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.443885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56dwj\" (UniqueName: \"kubernetes.io/projected/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-kube-api-access-56dwj\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.444216 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.443922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-metrics-tls\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.444643 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.444607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-tmp-dir\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.444871 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.444851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-config-volume\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.446291 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.446270 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-metrics-tls\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.466297 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.466269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:42:33.501462 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.501434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56dwj\" (UniqueName: \"kubernetes.io/projected/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-kube-api-access-56dwj\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.508467 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.508399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:33.537659 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.537621 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf"] Nov 01 03:42:33.540124 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.540108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.589444 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.589420 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 03:42:33.589444 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.589435 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 03:42:33.589600 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.589531 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 03:42:33.589928 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.589910 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 03:42:33.589928 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.589920 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w8pg8\"" Nov 01 03:42:33.590086 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.590006 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 03:42:33.596515 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.596497 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf"] Nov 01 03:42:33.645114 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.645086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.645264 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.645122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.645264 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.645170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfdx\" (UniqueName: \"kubernetes.io/projected/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-kube-api-access-fnfdx\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.645264 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.645248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.745688 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.745654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfdx\" (UniqueName: \"kubernetes.io/projected/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-kube-api-access-fnfdx\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.745836 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.745698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.745958 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.745940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.746000 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.745977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.746568 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.746546 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.748206 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.748189 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.748305 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.748287 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.803199 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.803138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfdx\" (UniqueName: \"kubernetes.io/projected/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-kube-api-access-fnfdx\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.830692 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.830546 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzs95"] Nov 01 03:42:33.833407 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:33.833192 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2cc2f4_e1a1_4da1_9591_ebeb958d4f66.slice/crio-62becf816bd512c417ea5154b84545444e20cc11702b8bb28dd56c1cfe3c5647 WatchSource:0}: Error finding container 62becf816bd512c417ea5154b84545444e20cc11702b8bb28dd56c1cfe3c5647: Status 404 returned error can't find the container with id 62becf816bd512c417ea5154b84545444e20cc11702b8bb28dd56c1cfe3c5647 Nov 01 03:42:33.844816 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.844793 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fvvjf"] Nov 01 03:42:33.848080 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:33.848055 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c97174_d65c_4516_923f_c504861b0196.slice/crio-cb05e997b4603abaaf58dadf4506c57db933d38e4730697385c984223d39fd3a WatchSource:0}: Error finding container cb05e997b4603abaaf58dadf4506c57db933d38e4730697385c984223d39fd3a: Status 404 returned error can't find the container with id cb05e997b4603abaaf58dadf4506c57db933d38e4730697385c984223d39fd3a Nov 01 03:42:33.852428 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.852411 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:42:33.878950 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.878925 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7kp4z"] Nov 01 03:42:33.882376 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:33.881986 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9727d7d4_17a6_4ae0_a3d9_61c3d8c877b7.slice/crio-4e4b2496ed0a5b0e53c97183c8ab28ee80817e1bb6155fa0f24dec6f9697c6b8 WatchSource:0}: Error finding container 4e4b2496ed0a5b0e53c97183c8ab28ee80817e1bb6155fa0f24dec6f9697c6b8: Status 404 returned error can't find the container with id 4e4b2496ed0a5b0e53c97183c8ab28ee80817e1bb6155fa0f24dec6f9697c6b8 Nov 01 03:42:33.966846 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.966800 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7kp4z" event={"ID":"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7","Type":"ContainerStarted","Data":"4e4b2496ed0a5b0e53c97183c8ab28ee80817e1bb6155fa0f24dec6f9697c6b8"} Nov 01 03:42:33.967987 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.967958 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fvvjf" event={"ID":"54c97174-d65c-4516-923f-c504861b0196","Type":"ContainerStarted","Data":"cb05e997b4603abaaf58dadf4506c57db933d38e4730697385c984223d39fd3a"} Nov 01 03:42:33.969441 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.969417 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzs95" event={"ID":"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66","Type":"ContainerStarted","Data":"64c7baef2eab97fd9c987ba27823345f333c9e3f8bb61a026185a35932159e3e"} Nov 01 03:42:33.969535 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:33.969474 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzs95" event={"ID":"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66","Type":"ContainerStarted","Data":"62becf816bd512c417ea5154b84545444e20cc11702b8bb28dd56c1cfe3c5647"} Nov 01 03:42:34.217469 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.217435 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf"] Nov 01 03:42:34.221956 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:34.221934 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1e0e21_7af3_4316_b343_bcc39d1dcb85.slice/crio-bec7ace172a9d7a6a50e0ebeefca2ecf8acd9775f6e98cfef685653a5c39cf12 WatchSource:0}: Error finding container bec7ace172a9d7a6a50e0ebeefca2ecf8acd9775f6e98cfef685653a5c39cf12: Status 404 returned error can't find the container with id bec7ace172a9d7a6a50e0ebeefca2ecf8acd9775f6e98cfef685653a5c39cf12 Nov 01 03:42:34.784816 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.784781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:34.784977 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.784795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:34.784977 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.784928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:34.834551 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.834522 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 03:42:34.880869 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.880847 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jzxr2\"" Nov 01 03:42:34.880992 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.880884 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 03:42:34.880992 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.880891 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v5dfc\"" Nov 01 03:42:34.881290 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.881276 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-69w79\"" Nov 01 03:42:34.881325 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.881290 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 03:42:34.881360 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.881350 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 03:42:34.972737 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:34.972712 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" event={"ID":"6f1e0e21-7af3-4316-b343-bcc39d1dcb85","Type":"ContainerStarted","Data":"bec7ace172a9d7a6a50e0ebeefca2ecf8acd9775f6e98cfef685653a5c39cf12"} Nov 01 03:42:35.976706 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:35.976671 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzs95" event={"ID":"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66","Type":"ContainerStarted","Data":"8a1333b70d721d1fc274b380903031be6c50035d6c9d2983afd512ff65b6f3c8"} Nov 01 03:42:36.483217 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.482763 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bd4ff9647-pldxz"] Nov 01 03:42:36.484903 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.484873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.534151 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.534109 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 03:42:36.567743 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.567900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.567900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbfv\" (UniqueName: \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.567900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.567900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.567900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.568746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.568746 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.567923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.580578 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.580548 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv2p7\"" Nov 01 03:42:36.580700 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.580596 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 03:42:36.580700 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.580662 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 03:42:36.580806 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.580758 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 03:42:36.580806 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.580774 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 03:42:36.580900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.580819 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 03:42:36.580900 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.580844 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 03:42:36.669106 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.669278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.669278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbfv\" (UniqueName: \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.669278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.669278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.669468 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.669468 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.669468 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.670016 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.669992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.670584 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.670562 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.671963 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.671941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.672510 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.672461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.675308 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.675288 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.675398 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.675328 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 03:42:36.675398 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.675385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.676605 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.676582 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd4ff9647-pldxz"] Nov 01 03:42:36.681977 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.681956 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.735521 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.735456 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbfv\" (UniqueName: \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:36.798395 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:36.798360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:37.191063 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:37.190835 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd4ff9647-pldxz"] Nov 01 03:42:37.983977 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:37.983937 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4ff9647-pldxz" event={"ID":"2fab48c4-836f-455c-9b44-f374e57f6079","Type":"ContainerStarted","Data":"1e080e297c85fb9ba3375b4d44266b365447bbb77d4e4ccab33a1f5ec2eebbce"} Nov 01 03:42:37.986224 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:37.986191 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7kp4z" event={"ID":"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7","Type":"ContainerStarted","Data":"535cc3113cb3abe575e46e84f22b6e9d2d347bf5fbc0247529bf3735b5166f19"} Nov 01 03:42:37.986349 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:37.986245 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7kp4z" event={"ID":"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7","Type":"ContainerStarted","Data":"48f81bf11acf62ec9a6c78d7b7296376763c099969cf5f5c6158ed883e42d364"} Nov 01 03:42:37.986396 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:37.986346 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:38.097212 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:38.097161 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7kp4z" podStartSLOduration=1.949942644 podStartE2EDuration="5.097144991s" totalImagesPullingTime="3.147202347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:38.096568664 +0000 UTC m=+44.867877887" watchObservedRunningTime="2026-11-01 03:42:38.097144991 +0000 UTC m=+44.868454204" Nov 01 03:42:38.993730 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:38.993477 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzs95" event={"ID":"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66","Type":"ContainerStarted","Data":"a798ede9854443c78e9d1fe5250fd06539a5c58e1dd19001b90b4f41bdf6b658"} Nov 01 03:42:38.996962 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:38.996920 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" event={"ID":"6f1e0e21-7af3-4316-b343-bcc39d1dcb85","Type":"ContainerStarted","Data":"2a7c157421293ed959c556679e0a3ebb72aec9f964631ca2c309209dfac6e797"} Nov 01 03:42:38.997108 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:38.996986 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" event={"ID":"6f1e0e21-7af3-4316-b343-bcc39d1dcb85","Type":"ContainerStarted","Data":"b9d91dbcd90a93f7aa134a2fafed17f20673f0a99c9ca8661ebcd1a4e9de297a"} Nov 01 03:42:39.105218 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:39.105157 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lzs95" podStartSLOduration=1.4146909060000001 podStartE2EDuration="6.105140229s" totalImagesPullingTime="4.690449323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:42:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:39.104301794 +0000 UTC m=+45.875611042" watchObservedRunningTime="2026-11-01 03:42:39.105140229 +0000 UTC m=+45.876449454" Nov 01 03:42:39.215198 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:39.215129 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" podStartSLOduration=2.098461494 podStartE2EDuration="6.215107779s" totalImagesPullingTime="4.116646285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:39.214442161 +0000 UTC m=+45.985751382" watchObservedRunningTime="2026-11-01 03:42:39.215107779 +0000 UTC m=+45.986417002" Nov 01 03:42:42.009206 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.009171 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fvvjf" event={"ID":"54c97174-d65c-4516-923f-c504861b0196","Type":"ContainerStarted","Data":"1f18c11bc9a667e207c5bb45b1e7cfbbc42631d5dd168ed4503cb3284711d4cb"} Nov 01 03:42:42.117935 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.117877 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fvvjf" podStartSLOduration=2.000408126 podStartE2EDuration="9.117861194s" totalImagesPullingTime="7.117453068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:42.11692676 +0000 UTC m=+48.888235983" watchObservedRunningTime="2026-11-01 03:42:42.117861194 +0000 UTC m=+48.889170413" Nov 01 03:42:42.164193 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.164163 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8"] Nov 01 03:42:42.186268 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.186243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.215702 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.215609 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8"] Nov 01 03:42:42.235354 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.235329 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 03:42:42.235500 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.235360 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 03:42:42.235500 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.235448 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hqffg\"" Nov 01 03:42:42.261758 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.261695 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r4gkc"] Nov 01 03:42:42.282570 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.282545 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs"] Nov 01 03:42:42.282692 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.282679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.290788 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.290766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.317850 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.317826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.317972 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.317855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9t7\" (UniqueName: \"kubernetes.io/projected/5cf75ba2-0374-460d-af57-81cebb276672-kube-api-access-hx9t7\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.317972 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.317883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.317972 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.317910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf75ba2-0374-460d-af57-81cebb276672-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.331914 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.331891 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 03:42:42.335285 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.335266 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 03:42:42.335381 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.335342 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 03:42:42.335436 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.335395 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ztkjl\"" Nov 01 03:42:42.339657 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.339640 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 03:42:42.339737 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.339643 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 03:42:42.339737 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.339672 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 03:42:42.339737 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.339728 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v6qjk\"" Nov 01 03:42:42.378361 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.378337 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs"] Nov 01 03:42:42.419168 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-metrics-client-ca\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.419278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-sys\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419349 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-wtmp\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419349 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.419423 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-textfile\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419469 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf75ba2-0374-460d-af57-81cebb276672-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.419518 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-root\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419518 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/34608ae4-4158-4e76-88bd-bdb4d1381f87-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.419612 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmhn\" (UniqueName: \"kubernetes.io/projected/07e58839-259c-453a-98f1-dac051fb8f9e-kube-api-access-zkmhn\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419612 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.419739 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419780 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9t7\" (UniqueName: \"kubernetes.io/projected/5cf75ba2-0374-460d-af57-81cebb276672-kube-api-access-hx9t7\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.419822 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.419822 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmc2q\" (UniqueName: \"kubernetes.io/projected/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-api-access-bmc2q\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.419897 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.419897 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.419964 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.419964 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.419938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-tls\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.422855 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.422834 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf75ba2-0374-460d-af57-81cebb276672-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.427486 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.427465 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.427486 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.427478 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.477308 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.477277 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9t7\" (UniqueName: \"kubernetes.io/projected/5cf75ba2-0374-460d-af57-81cebb276672-kube-api-access-hx9t7\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.521094 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.520991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.521278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmc2q\" (UniqueName: \"kubernetes.io/projected/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-api-access-bmc2q\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.521278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.521278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-tls\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521278 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-metrics-client-ca\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-sys\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-wtmp\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-textfile\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-root\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/34608ae4-4158-4e76-88bd-bdb4d1381f87-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.521685 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.521648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmhn\" (UniqueName: \"kubernetes.io/projected/07e58839-259c-453a-98f1-dac051fb8f9e-kube-api-access-zkmhn\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.523630 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.522130 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-root\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.523630 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.522863 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-wtmp\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.523630 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.522975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-sys\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.523630 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.523258 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.523630 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.523392 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.529777 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.528272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/34608ae4-4158-4e76-88bd-bdb4d1381f87-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.530159 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.530107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.530408 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.530386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-textfile\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.531223 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.531173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.531549 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.531486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.532192 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.532167 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.532292 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.532269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-metrics-client-ca\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.532588 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.532553 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-tls\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.569463 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.569435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:42:42.579328 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.579306 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmc2q\" (UniqueName: \"kubernetes.io/projected/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-api-access-bmc2q\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:42.579639 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.579623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmhn\" (UniqueName: \"kubernetes.io/projected/07e58839-259c-453a-98f1-dac051fb8f9e-kube-api-access-zkmhn\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.623134 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.623107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:42:42.656070 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:42.654154 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e58839_259c_453a_98f1_dac051fb8f9e.slice/crio-77309725d225f5c436ce0c905317d57497ae331ecd2d652d94bd1ce6282e55cf WatchSource:0}: Error finding container 77309725d225f5c436ce0c905317d57497ae331ecd2d652d94bd1ce6282e55cf: Status 404 returned error can't find the container with id 77309725d225f5c436ce0c905317d57497ae331ecd2d652d94bd1ce6282e55cf Nov 01 03:42:42.656414 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:42.656301 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:42:43.007117 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:43.007088 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8"] Nov 01 03:42:43.013909 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:43.013873 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4ff9647-pldxz" event={"ID":"2fab48c4-836f-455c-9b44-f374e57f6079","Type":"ContainerStarted","Data":"5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d"} Nov 01 03:42:43.014997 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:43.014970 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4gkc" event={"ID":"07e58839-259c-453a-98f1-dac051fb8f9e","Type":"ContainerStarted","Data":"77309725d225f5c436ce0c905317d57497ae331ecd2d652d94bd1ce6282e55cf"} Nov 01 03:42:43.047697 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:43.047671 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf75ba2_0374_460d_af57_81cebb276672.slice/crio-c5355dddfa638fd8b4c77bbd82acbaeceee8da7e12d2eb5307de664d6dc7dd64 WatchSource:0}: Error finding container c5355dddfa638fd8b4c77bbd82acbaeceee8da7e12d2eb5307de664d6dc7dd64: Status 404 returned error can't find the container with id c5355dddfa638fd8b4c77bbd82acbaeceee8da7e12d2eb5307de664d6dc7dd64 Nov 01 03:42:43.062699 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:43.062673 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs"] Nov 01 03:42:43.070917 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:43.070895 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34608ae4_4158_4e76_88bd_bdb4d1381f87.slice/crio-d3f87f4d397e00d4abb66aed03b6ce7f6dc7ee1022c8eee4377c18fd89878bc2 WatchSource:0}: Error finding container d3f87f4d397e00d4abb66aed03b6ce7f6dc7ee1022c8eee4377c18fd89878bc2: Status 404 returned error can't find the container with id d3f87f4d397e00d4abb66aed03b6ce7f6dc7ee1022c8eee4377c18fd89878bc2 Nov 01 03:42:43.131908 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:43.131849 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bd4ff9647-pldxz" podStartSLOduration=1.935562412 podStartE2EDuration="7.131832131s" totalImagesPullingTime="5.196269719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:43.130706083 +0000 UTC m=+49.902015305" watchObservedRunningTime="2026-11-01 03:42:43.131832131 +0000 UTC m=+49.903141354" Nov 01 03:42:44.019998 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:44.019964 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" event={"ID":"34608ae4-4158-4e76-88bd-bdb4d1381f87","Type":"ContainerStarted","Data":"d3f87f4d397e00d4abb66aed03b6ce7f6dc7ee1022c8eee4377c18fd89878bc2"} Nov 01 03:42:44.022095 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:44.022065 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" event={"ID":"5cf75ba2-0374-460d-af57-81cebb276672","Type":"ContainerStarted","Data":"7574f8c613864c5006ad09a3abdaf2c6c656fc6abb6d2ba95d50345355116160"} Nov 01 03:42:44.022238 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:44.022129 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" event={"ID":"5cf75ba2-0374-460d-af57-81cebb276672","Type":"ContainerStarted","Data":"4a9a146f7e9635a25deacedcf152d6884c8a13d8226b00e57f2d69cc645294a0"} Nov 01 03:42:44.022238 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:44.022160 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" event={"ID":"5cf75ba2-0374-460d-af57-81cebb276672","Type":"ContainerStarted","Data":"c5355dddfa638fd8b4c77bbd82acbaeceee8da7e12d2eb5307de664d6dc7dd64"} Nov 01 03:42:45.031420 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:45.031375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4gkc" event={"ID":"07e58839-259c-453a-98f1-dac051fb8f9e","Type":"ContainerStarted","Data":"cf3d914e5f0cc66b100c2118c823bd532046b0171559289ea45423a02c577ed7"} Nov 01 03:42:46.036413 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:46.036369 2579 generic.go:356] "Generic (PLEG): container finished" podID="07e58839-259c-453a-98f1-dac051fb8f9e" containerID="cf3d914e5f0cc66b100c2118c823bd532046b0171559289ea45423a02c577ed7" exitCode=0 Nov 01 03:42:46.036413 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:46.036421 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4gkc" event={"ID":"07e58839-259c-453a-98f1-dac051fb8f9e","Type":"ContainerDied","Data":"cf3d914e5f0cc66b100c2118c823bd532046b0171559289ea45423a02c577ed7"} Nov 01 03:42:46.799213 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:46.799175 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:46.799414 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:46.799246 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:46.804012 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:46.803990 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:47.041799 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.041758 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" event={"ID":"5cf75ba2-0374-460d-af57-81cebb276672","Type":"ContainerStarted","Data":"d7d106f0d7f1a867b119b5624b5f346eb83aef9dcb5faaf92cc50beb3af5dc8e"} Nov 01 03:42:47.043997 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.043967 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4gkc" event={"ID":"07e58839-259c-453a-98f1-dac051fb8f9e","Type":"ContainerStarted","Data":"e83f7ad56557aa76f6f2f2184831e7dbb168b7ddbf80b5f519c6d9cb857189ea"} Nov 01 03:42:47.044161 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.044011 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4gkc" event={"ID":"07e58839-259c-453a-98f1-dac051fb8f9e","Type":"ContainerStarted","Data":"a6bdcec0a3742e4f7008e0c9cca4701b922dd5f5826b14dbe056da0c6ea42970"} Nov 01 03:42:47.046296 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.046263 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" event={"ID":"34608ae4-4158-4e76-88bd-bdb4d1381f87","Type":"ContainerStarted","Data":"7c45e9a5ac0fbbb9c62897be4f9eaca68cc489c22ddd890515b5476c9e9f767e"} Nov 01 03:42:47.046409 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.046311 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" event={"ID":"34608ae4-4158-4e76-88bd-bdb4d1381f87","Type":"ContainerStarted","Data":"58131f31b97cf0861d453bb0c401d57a440add0d0f42df8b8b01ea55401a7723"} Nov 01 03:42:47.046409 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.046332 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" event={"ID":"34608ae4-4158-4e76-88bd-bdb4d1381f87","Type":"ContainerStarted","Data":"576d04bae8b9eea117a8affb2aafff1ec2d72a0d230fc0d584dfa067e3239161"} Nov 01 03:42:47.050326 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.050276 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:42:47.154982 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.154922 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" podStartSLOduration=2.1843189020000002 podStartE2EDuration="5.15490094s" totalImagesPullingTime="2.970582038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:47.153595989 +0000 UTC m=+53.924905210" watchObservedRunningTime="2026-11-01 03:42:47.15490094 +0000 UTC m=+53.926210164" Nov 01 03:42:47.379185 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.379077 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" podStartSLOduration=2.241373115 podStartE2EDuration="5.379059654s" totalImagesPullingTime="3.137686539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:47.37824106 +0000 UTC m=+54.149550282" watchObservedRunningTime="2026-11-01 03:42:47.379059654 +0000 UTC m=+54.150368903" Nov 01 03:42:47.490981 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.490920 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r4gkc" podStartSLOduration=3.174999118 podStartE2EDuration="5.490900603s" totalImagesPullingTime="2.225490506s" totalInitContainerRuntime="90.410979ms" isStatefulPod=true podCreationTimestamp="2026-11-01 03:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:47.489857456 +0000 UTC m=+54.261166678" watchObservedRunningTime="2026-11-01 03:42:47.490900603 +0000 UTC m=+54.262209825" Nov 01 03:42:47.918347 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.918317 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt"] Nov 01 03:42:47.920644 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.920623 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967077 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967234 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-federate-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967234 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967234 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967372 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967372 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-metrics-client-ca\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967440 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-serving-certs-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.967440 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.967431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rt5\" (UniqueName: \"kubernetes.io/projected/ab862706-1c0c-4324-86d7-59c9b6313672-kube-api-access-r5rt5\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:47.969630 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.969611 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 03:42:47.969825 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.969806 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 03:42:47.998654 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:47.998630 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7kp4z" Nov 01 03:42:48.016263 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.016242 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 03:42:48.016263 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.016258 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 03:42:48.016444 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.016335 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 03:42:48.016550 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.016475 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zddqk\"" Nov 01 03:42:48.063231 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.063202 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 03:42:48.064411 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.064394 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt"] Nov 01 03:42:48.068453 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.068434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-federate-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.068525 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.068470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.068525 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.068499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.068638 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.068598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.068687 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.068659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-metrics-client-ca\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.069086 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.069024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-serving-certs-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.069177 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.069141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rt5\" (UniqueName: \"kubernetes.io/projected/ab862706-1c0c-4324-86d7-59c9b6313672-kube-api-access-r5rt5\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.069589 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.069552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.069804 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.069777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-serving-certs-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.069969 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.069942 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.070826 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.070802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-metrics-client-ca\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.071541 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.071520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-federate-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.071635 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.071548 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.071635 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.071566 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.072320 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.072302 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.130657 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.130629 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rt5\" (UniqueName: \"kubernetes.io/projected/ab862706-1c0c-4324-86d7-59c9b6313672-kube-api-access-r5rt5\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.233356 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.233319 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:42:48.646855 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:48.646778 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt"] Nov 01 03:42:48.649645 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:48.649618 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab862706_1c0c_4324_86d7_59c9b6313672.slice/crio-4b3fccaca1ac6b70542db96f4dc40b7e60cf554e3717387803f741cbdacb2979 WatchSource:0}: Error finding container 4b3fccaca1ac6b70542db96f4dc40b7e60cf554e3717387803f741cbdacb2979: Status 404 returned error can't find the container with id 4b3fccaca1ac6b70542db96f4dc40b7e60cf554e3717387803f741cbdacb2979 Nov 01 03:42:49.056504 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:49.056472 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" event={"ID":"ab862706-1c0c-4324-86d7-59c9b6313672","Type":"ContainerStarted","Data":"4b3fccaca1ac6b70542db96f4dc40b7e60cf554e3717387803f741cbdacb2979"} Nov 01 03:42:50.949212 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:50.949180 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:42:53.074263 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:53.074224 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" event={"ID":"ab862706-1c0c-4324-86d7-59c9b6313672","Type":"ContainerStarted","Data":"c5c7d16ce464191482809bd770705ec71ee642192c8af3d2b809f371ba99b70e"} Nov 01 03:42:55.085271 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:55.085229 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" event={"ID":"ab862706-1c0c-4324-86d7-59c9b6313672","Type":"ContainerStarted","Data":"807553f58bc17ecffc52bc4a9331af67d819265ca6f9ee014ba188537b6ae385"} Nov 01 03:42:55.085675 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:55.085287 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" event={"ID":"ab862706-1c0c-4324-86d7-59c9b6313672","Type":"ContainerStarted","Data":"11db9a4f21f1e5178810de7733460ba531c3086cab8bc52ce1f727885952050b"} Nov 01 03:42:55.297866 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:55.297811 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" podStartSLOduration=2.768161709 podStartE2EDuration="8.29779821s" totalImagesPullingTime="5.529636501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:55.250881387 +0000 UTC m=+62.022190608" watchObservedRunningTime="2026-11-01 03:42:55.29779821 +0000 UTC m=+62.069107432" Nov 01 03:42:58.565166 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.565135 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d996f7-5ldfz"] Nov 01 03:42:58.567153 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.567134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.625635 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.625603 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d996f7-5ldfz"] Nov 01 03:42:58.691769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.691732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.691769 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.691765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.692005 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.691795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.692005 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.691820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.692005 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.691982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62227\" (UniqueName: \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.692169 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.692017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.692169 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.692104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.692241 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.692163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793399 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793399 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793602 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62227\" (UniqueName: \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793602 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793602 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793602 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793602 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793602 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.793934 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.793909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.794438 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.794409 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.794527 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.794443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.794527 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.794480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.794617 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.794593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.796094 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.796067 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.796175 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.796067 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.850584 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.850529 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62227\" (UniqueName: \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:58.877421 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:58.877393 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:42:59.245288 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.245259 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d996f7-5ldfz"] Nov 01 03:42:59.248164 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:42:59.248137 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc030034_30dc_4b76_836d_c0516aa54384.slice/crio-51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de WatchSource:0}: Error finding container 51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de: Status 404 returned error can't find the container with id 51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de Nov 01 03:42:59.701321 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.701241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:59.701321 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.701298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:59.749373 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.749341 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 03:42:59.754770 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.754744 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:42:59.795832 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.795800 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 03:42:59.802245 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.802216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:42:59.804431 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.804413 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:42:59.850826 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.850803 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 03:42:59.900718 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.900697 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 03:42:59.905508 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:42:59.905485 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:43:00.044424 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.044393 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jzxr2\"" Nov 01 03:43:00.046471 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.046453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:43:00.052377 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.052346 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v5dfc\"" Nov 01 03:43:00.057917 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.057893 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-69w79\"" Nov 01 03:43:00.058910 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.058888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:43:00.062971 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.062948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:00.104514 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.104469 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d996f7-5ldfz" event={"ID":"fc030034-30dc-4b76-836d-c0516aa54384","Type":"ContainerStarted","Data":"39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1"} Nov 01 03:43:00.104700 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.104525 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d996f7-5ldfz" event={"ID":"fc030034-30dc-4b76-836d-c0516aa54384","Type":"ContainerStarted","Data":"51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de"} Nov 01 03:43:00.218122 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.218076 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56d996f7-5ldfz" podStartSLOduration=2.218019325 podStartE2EDuration="2.218019325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:43:00.216802937 +0000 UTC m=+66.988112158" watchObservedRunningTime="2026-11-01 03:43:00.218019325 +0000 UTC m=+66.989328546" Nov 01 03:43:00.442368 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.442292 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r4j7n"] Nov 01 03:43:00.483665 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.483562 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nwbf4"] Nov 01 03:43:00.484737 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:00.484718 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h7fxk"] Nov 01 03:43:00.485821 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:43:00.485794 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128c7b38_4784_403c_be55_e2f1561c2f72.slice/crio-06766e8b91d6988aa5a5fd028a4501bf8670733cae6bc10a5489538d61cbb940 WatchSource:0}: Error finding container 06766e8b91d6988aa5a5fd028a4501bf8670733cae6bc10a5489538d61cbb940: Status 404 returned error can't find the container with id 06766e8b91d6988aa5a5fd028a4501bf8670733cae6bc10a5489538d61cbb940 Nov 01 03:43:00.490833 ip-10-0-128-69 kubenswrapper[2579]: W1101 03:43:00.490804 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6cbdc5_6506_47c1_a44e_6d185d3fc716.slice/crio-0ea4e7a2975a1ebcaa8c2f702cfb9e1226e6d22f82b3adc3d0a93debfcbcaf94 WatchSource:0}: Error finding container 0ea4e7a2975a1ebcaa8c2f702cfb9e1226e6d22f82b3adc3d0a93debfcbcaf94: Status 404 returned error can't find the container with id 0ea4e7a2975a1ebcaa8c2f702cfb9e1226e6d22f82b3adc3d0a93debfcbcaf94 Nov 01 03:43:01.109438 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:01.109398 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nwbf4" event={"ID":"128c7b38-4784-403c-be55-e2f1561c2f72","Type":"ContainerStarted","Data":"06766e8b91d6988aa5a5fd028a4501bf8670733cae6bc10a5489538d61cbb940"} Nov 01 03:43:01.111317 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:01.111286 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r4j7n" event={"ID":"8d064689-ea69-4221-b860-9242ee3f80a0","Type":"ContainerStarted","Data":"36281239ffcf777e5a4afa756d27d386564fea92903f5f5e061ddc15f4e54d63"} Nov 01 03:43:01.112552 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:01.112528 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7fxk" event={"ID":"df6cbdc5-6506-47c1-a44e-6d185d3fc716","Type":"ContainerStarted","Data":"0ea4e7a2975a1ebcaa8c2f702cfb9e1226e6d22f82b3adc3d0a93debfcbcaf94"} Nov 01 03:43:03.122703 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:03.122662 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7fxk" event={"ID":"df6cbdc5-6506-47c1-a44e-6d185d3fc716","Type":"ContainerStarted","Data":"1a61a32f8ed95548a10e85308c0f2c453e6b46a57ad8d73615478d1427ff1e08"} Nov 01 03:43:03.123229 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:03.122735 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h7fxk" event={"ID":"df6cbdc5-6506-47c1-a44e-6d185d3fc716","Type":"ContainerStarted","Data":"167420d38b7975073305e1efdec0bc301dde4fa29aae2dd4682b14d49af735f7"} Nov 01 03:43:03.234729 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:03.234663 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h7fxk" podStartSLOduration=68.038710319 podStartE2EDuration="1m10.234647743s" totalImagesPullingTime="2.195937424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:43:03.233779607 +0000 UTC m=+70.005088828" watchObservedRunningTime="2026-11-01 03:43:03.234647743 +0000 UTC m=+70.005956964" Nov 01 03:43:03.452964 ip-10-0-128-69 kubenswrapper[2579]: E1101 03:43:03.452880 2579 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:43:05.131903 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:05.131868 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r4j7n" event={"ID":"8d064689-ea69-4221-b860-9242ee3f80a0","Type":"ContainerStarted","Data":"e0197068ff53b15aef0ba25ad3047e834ae390521987a44987390fa9ff62b15d"} Nov 01 03:43:05.132391 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:05.132061 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:43:05.245715 ip-10-0-128-69 kubenswrapper[2579]: I1101 03:43:05.245654 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r4j7n" podStartSLOduration=67.740798107 podStartE2EDuration="1m12.245631632s" totalImagesPullingTime="4.504833525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:43:05.244792105 +0000 UTC m=+72.016101328" watchObservedRunningTime="2026-11-01 03:43:05.245631632 +0000 UTC m=+72.016940854" Nov 01 03:43:08.350139 ip-10-0-128-69 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 03:43:08.364473 ip-10-0-128-69 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 03:43:08.364689 ip-10-0-128-69 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 03:43:08.364888 ip-10-0-128-69 systemd[1]: kubelet.service: Consumed 4.122s CPU time. Nov 01 03:43:08.379796 ip-10-0-128-69 systemd[1]: Starting Kubernetes Kubelet... Nov 01 03:43:08.439804 ip-10-0-128-69 kubenswrapper[6735]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 03:43:08.439804 ip-10-0-128-69 kubenswrapper[6735]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 03:43:08.439804 ip-10-0-128-69 kubenswrapper[6735]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 03:43:08.439804 ip-10-0-128-69 kubenswrapper[6735]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 03:43:08.442025 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.441948 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.442025 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442025 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442028 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442049 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442054 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442059 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442063 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442066 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442070 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442073 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442075 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442084 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442088 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442090 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442093 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442095 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442098 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442100 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442103 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442106 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.442118 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442108 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442111 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442113 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442116 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442118 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442120 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442123 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442128 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442132 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442135 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442138 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442141 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442144 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442146 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442149 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442151 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442154 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442156 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442158 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.442588 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442161 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442163 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442166 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442168 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442171 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442173 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442175 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442178 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442180 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442183 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442185 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442188 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442190 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442193 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442196 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442217 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442221 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442224 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442227 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.443366 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442229 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442232 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442235 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442238 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442241 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442243 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442246 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442250 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442253 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442256 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442259 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442262 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442264 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442267 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442270 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442272 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442275 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442277 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442280 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.443849 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442282 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442286 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442289 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442292 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442294 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442297 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442300 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442303 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442305 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442308 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442311 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442314 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442316 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442319 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442321 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442325 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442327 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442330 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442332 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442334 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.444322 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442337 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442340 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442343 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442345 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442348 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442351 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442353 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442356 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.442359 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444488 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444500 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444504 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444507 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444510 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444513 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444516 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444518 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444521 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444523 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444527 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.444856 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444529 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444532 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444534 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444537 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444540 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444543 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444545 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444548 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444550 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444554 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444558 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444561 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444564 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444567 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444570 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444573 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444575 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444578 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444581 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.445375 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444584 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444587 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444591 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444594 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444597 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444599 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444602 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444604 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444607 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444610 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444612 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444615 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444621 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444624 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444626 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444629 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444633 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444637 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444640 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.445833 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444642 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444645 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444647 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444650 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444652 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444655 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444658 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444660 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444662 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444665 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444667 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444670 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444672 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444676 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444679 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444681 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444684 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444686 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444689 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444691 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.446325 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444694 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444696 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444698 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444701 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444704 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444708 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444711 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444713 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444716 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444719 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444721 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444724 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444726 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444729 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444732 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444734 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444737 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444739 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444742 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.446852 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444744 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444747 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444749 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444753 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444755 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444758 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444760 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444763 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444766 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444768 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444770 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444773 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444776 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444778 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444780 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444783 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444785 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.444788 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445337 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.447336 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445345 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445349 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445351 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445354 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445357 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445360 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445362 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445365 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445369 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445372 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445374 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445377 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445380 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445383 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445386 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445388 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445391 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445393 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445397 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.447808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445399 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445402 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445404 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445407 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445409 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445411 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445414 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445418 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445421 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445424 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445427 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445429 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445433 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445435 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445438 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445440 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445443 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445445 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445448 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445450 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.448293 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445453 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445455 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445459 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445461 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445464 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445466 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445469 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445471 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445473 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445476 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445478 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445482 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445485 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445487 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445489 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445492 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445494 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445497 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445499 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445501 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.448792 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445504 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445506 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445509 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445511 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445513 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445518 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445521 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445524 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445527 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445530 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445532 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445535 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445537 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445540 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445542 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445545 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445547 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445550 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445552 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.449316 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445555 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445557 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445560 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445562 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445565 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445568 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445571 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445573 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445576 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445578 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445581 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445583 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445586 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445588 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445591 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445593 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445595 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445598 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445600 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445603 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.449773 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445605 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445607 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445610 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445612 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445616 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445618 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.445621 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445710 6735 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445717 6735 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445724 6735 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445729 6735 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445733 6735 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445736 6735 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445740 6735 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445744 6735 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445747 6735 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445751 6735 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445754 6735 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445757 6735 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445760 6735 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445763 6735 flags.go:64] FLAG: --cgroup-root="" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445766 6735 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 03:43:08.450258 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445769 6735 flags.go:64] FLAG: --client-ca-file="" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445771 6735 flags.go:64] FLAG: --cloud-provider="external" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445774 6735 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445778 6735 flags.go:64] FLAG: --cluster-domain="" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445784 6735 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445787 6735 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445790 6735 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445793 6735 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445800 6735 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445803 6735 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445806 6735 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445809 6735 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445812 6735 flags.go:64] FLAG: --contention-profiling="false" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445815 6735 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445818 6735 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445821 6735 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445824 6735 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445829 6735 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445832 6735 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445835 6735 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445838 6735 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445842 6735 flags.go:64] FLAG: --enable-server="true" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445844 6735 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445849 6735 flags.go:64] FLAG: --event-burst="100" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445852 6735 flags.go:64] FLAG: --event-qps="50" Nov 01 03:43:08.450827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445855 6735 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445858 6735 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445861 6735 flags.go:64] FLAG: --eviction-hard="" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445865 6735 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445868 6735 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445871 6735 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445874 6735 flags.go:64] FLAG: --eviction-soft="" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445878 6735 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445881 6735 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445884 6735 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445887 6735 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445893 6735 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445895 6735 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445898 6735 flags.go:64] FLAG: --feature-gates="" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445903 6735 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445906 6735 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445909 6735 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445912 6735 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445915 6735 flags.go:64] FLAG: --healthz-port="10248" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445918 6735 flags.go:64] FLAG: --help="false" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445921 6735 flags.go:64] FLAG: --hostname-override="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445924 6735 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445927 6735 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 03:43:08.451509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445930 6735 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445933 6735 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445937 6735 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445940 6735 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445942 6735 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445946 6735 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445949 6735 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445952 6735 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445955 6735 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445957 6735 flags.go:64] FLAG: --kube-reserved="" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445960 6735 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445963 6735 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445966 6735 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445969 6735 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445972 6735 flags.go:64] FLAG: --lock-file="" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445975 6735 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445978 6735 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445981 6735 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445987 6735 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445989 6735 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445994 6735 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.445997 6735 flags.go:64] FLAG: --logging-format="text" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446000 6735 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446003 6735 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 03:43:08.452068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446006 6735 flags.go:64] FLAG: --manifest-url="" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446009 6735 flags.go:64] FLAG: --manifest-url-header="" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446013 6735 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446016 6735 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446020 6735 flags.go:64] FLAG: --max-pods="110" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446023 6735 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446026 6735 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446029 6735 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446061 6735 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446064 6735 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446067 6735 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446070 6735 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446076 6735 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446079 6735 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446083 6735 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446086 6735 flags.go:64] FLAG: --pod-cidr="" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446091 6735 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446093 6735 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446096 6735 flags.go:64] FLAG: --pods-per-core="0" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446099 6735 flags.go:64] FLAG: --port="10250" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446102 6735 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446105 6735 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0cb39e7e1d45a7d12" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446108 6735 flags.go:64] FLAG: --qos-reserved="" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446111 6735 flags.go:64] FLAG: --read-only-port="10255" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446114 6735 flags.go:64] FLAG: --register-node="true" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446117 6735 flags.go:64] FLAG: --register-with-taints="" Nov 01 03:43:08.452647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446121 6735 flags.go:64] FLAG: --registry-burst="10" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446124 6735 flags.go:64] FLAG: --registry-qps="5" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446128 6735 flags.go:64] FLAG: --reserved-cpus="" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446131 6735 flags.go:64] FLAG: --reserved-memory="" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446135 6735 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446138 6735 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446141 6735 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446144 6735 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446147 6735 flags.go:64] FLAG: --runonce="false" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446149 6735 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446152 6735 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446155 6735 flags.go:64] FLAG: --seccomp-default="false" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446158 6735 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446161 6735 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446164 6735 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446167 6735 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446170 6735 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446173 6735 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446176 6735 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446179 6735 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446182 6735 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446185 6735 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446192 6735 flags.go:64] FLAG: --system-cgroups="" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446195 6735 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446201 6735 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 03:43:08.453290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446204 6735 flags.go:64] FLAG: --tls-cert-file="" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446206 6735 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446211 6735 flags.go:64] FLAG: --tls-min-version="" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446213 6735 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446217 6735 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446220 6735 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446223 6735 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446225 6735 flags.go:64] FLAG: --v="2" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446230 6735 flags.go:64] FLAG: --version="false" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446234 6735 flags.go:64] FLAG: --vmodule="" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446238 6735 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446242 6735 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446389 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446392 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446395 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446398 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446401 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446404 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446406 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446409 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446411 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446414 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.453896 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446416 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446419 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446421 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446424 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446426 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446429 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446432 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446436 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446439 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446441 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446444 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446447 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446449 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446451 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446454 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446456 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446459 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446461 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446464 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446468 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.454461 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446470 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446474 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446478 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446480 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446484 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446486 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446489 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446491 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446493 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446496 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446498 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446501 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446503 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446506 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446508 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446510 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446513 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446515 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446517 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446521 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.454942 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446524 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446527 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446529 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446532 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446534 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446537 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446539 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446542 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446544 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446547 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446549 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446553 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446556 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446559 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446561 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446564 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446566 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446569 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446572 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446574 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.455436 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446576 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446579 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446581 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446585 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446589 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446592 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446595 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446597 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446600 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446602 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446605 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446608 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446611 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446614 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446617 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446619 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446621 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446624 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446627 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.455916 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446629 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446632 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446634 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446636 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446640 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446643 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446645 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446648 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446650 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446653 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446655 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446657 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446660 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446663 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446665 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446667 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.456446 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446670 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.446695 6735 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446738 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446743 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446746 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446748 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446751 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446754 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446758 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446761 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446763 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446769 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446772 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446774 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446777 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446779 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.457223 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446782 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446784 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446787 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446789 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446792 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446796 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446798 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446801 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446803 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446806 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446808 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446811 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446814 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446816 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446819 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446821 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446824 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446826 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446829 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446831 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.457832 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446834 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446836 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446840 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446844 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446848 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446851 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446853 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446856 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446859 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446863 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446865 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446868 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446870 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446873 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446875 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446878 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446881 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446885 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446887 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.458567 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446890 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446893 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446896 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446898 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446901 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446903 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446906 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446908 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446911 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446913 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446916 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446918 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446921 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446923 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446925 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446928 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446930 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446934 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446936 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.459150 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446938 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446941 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446943 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446947 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446949 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446952 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446954 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446957 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446959 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446961 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446964 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446967 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446970 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446973 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446975 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446978 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446980 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446983 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446986 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446988 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.459612 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446991 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446993 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446996 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.446998 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447001 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447003 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447006 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447009 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447014 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447018 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447022 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447025 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447027 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.447029 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.450990 6735 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.451007 6735 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.451025 6735 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.451030 6735 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451098 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.460164 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451105 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451109 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451113 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451117 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451120 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451122 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451125 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451129 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451133 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451136 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451138 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451141 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451144 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451147 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451150 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451152 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451155 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451157 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451160 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.460774 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451162 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451165 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451169 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451174 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451177 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451180 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451183 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451185 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451188 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451191 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451194 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451196 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451199 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451201 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451204 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451206 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451208 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451211 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451214 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.461303 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451217 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451220 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451222 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451225 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451228 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451230 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451233 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451235 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451238 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451241 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451244 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451246 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451249 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451251 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451254 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451256 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451259 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451262 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451265 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451269 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.461753 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451273 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451278 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451282 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451288 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451293 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451297 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451301 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451304 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451308 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451313 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451316 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451318 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451321 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451323 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451326 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451328 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451331 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451334 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451337 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.462256 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451339 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451342 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451345 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451347 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451350 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451352 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451355 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451357 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451360 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451362 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451365 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451367 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451370 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451372 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451374 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451377 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451380 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451383 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451385 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.462704 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451388 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451391 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451393 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451396 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451398 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451400 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451403 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451406 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451408 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.451437 6735 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451581 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451585 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451588 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451590 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451593 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451595 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.463253 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451598 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451600 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451604 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451608 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451611 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451614 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451617 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451620 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451623 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451625 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451628 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451630 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451633 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451635 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451637 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451642 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451645 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451647 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451650 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.463668 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451652 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451655 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451657 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451660 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451662 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451664 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451667 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451670 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451673 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451675 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451677 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451681 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451683 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451686 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451688 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451691 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451693 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451695 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451698 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451700 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.464133 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451703 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451705 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451708 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451710 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451712 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451715 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451717 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451720 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451722 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451725 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451728 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451730 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451733 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451735 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451737 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451740 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451742 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451745 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451747 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451750 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.464605 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451752 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451755 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451757 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451761 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451765 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451768 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451771 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451774 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451776 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451778 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451781 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451784 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451786 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451788 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451791 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451793 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451796 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451798 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451801 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.465119 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451803 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451805 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451808 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451811 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451814 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451816 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451818 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451821 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451823 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451825 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451828 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451830 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451833 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451836 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451838 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451841 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451843 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451846 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451848 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451850 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.465663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451853 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451855 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.451879 6735 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451911 6735 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451914 6735 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451917 6735 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451919 6735 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451922 6735 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451924 6735 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451927 6735 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451929 6735 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451931 6735 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451934 6735 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451936 6735 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451939 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451941 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 03:43:08.466166 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451944 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451947 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451949 6735 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451952 6735 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451954 6735 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451960 6735 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451964 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451966 6735 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451969 6735 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451972 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451975 6735 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451978 6735 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451980 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451983 6735 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451985 6735 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451988 6735 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451990 6735 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451992 6735 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451995 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 03:43:08.466544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.451997 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452000 6735 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452002 6735 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452005 6735 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452007 6735 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452010 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452012 6735 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452015 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452017 6735 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452020 6735 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452022 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452025 6735 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452027 6735 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452044 6735 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452050 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452052 6735 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452055 6735 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452057 6735 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452063 6735 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452065 6735 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 03:43:08.466994 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452068 6735 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452071 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452074 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452077 6735 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452079 6735 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452082 6735 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452085 6735 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452088 6735 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452090 6735 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452093 6735 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452095 6735 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452098 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452100 6735 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452103 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452105 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452108 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452110 6735 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452112 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452115 6735 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452117 6735 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 03:43:08.467474 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452120 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452122 6735 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452125 6735 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452127 6735 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452129 6735 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452132 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452136 6735 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452140 6735 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452143 6735 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452146 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452151 6735 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452154 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452157 6735 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452161 6735 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452163 6735 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452166 6735 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452169 6735 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452171 6735 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452173 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 03:43:08.467944 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452176 6735 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452178 6735 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452181 6735 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452183 6735 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452186 6735 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452188 6735 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452191 6735 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452193 6735 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452196 6735 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452198 6735 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452200 6735 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452203 6735 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452205 6735 feature_gate.go:386] unrecognized feature gate: Example Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452208 6735 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:08.452210 6735 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.452284 6735 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.453283 6735 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.453338 6735 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.453519 6735 server.go:1014] "Starting client certificate rotation" Nov 01 03:43:08.468407 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.453664 6735 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.453710 6735 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:36:53 +0000 UTC" deadline="2028-07-27 02:22:58.543902732 +0000 UTC" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.453759 6735 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15214h39m50.090146568s" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.454053 6735 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.455208 6735 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.458232 6735 log.go:25] "Validated CRI v1 runtime API" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.460050 6735 log.go:25] "Validated CRI v1 image API" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.460801 6735 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 03:43:08.468942 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.462813 6735 fs.go:135] Filesystem UUIDs: map[10401c2b-3bae-4dd2-86b7-509d4d73f4ef:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d5290e1b-521a-4723-8947-cc279d96ed3e:/dev/nvme0n1p4] Nov 01 03:43:08.469209 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.462836 6735 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0074608fcbb04640a9f12e496d37ecf08fe99646cce29b5ee6dc19098bcef04a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0074608fcbb04640a9f12e496d37ecf08fe99646cce29b5ee6dc19098bcef04a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06766e8b91d6988aa5a5fd028a4501bf8670733cae6bc10a5489538d61cbb940/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06766e8b91d6988aa5a5fd028a4501bf8670733cae6bc10a5489538d61cbb940/userdata/shm major:0 minor:389 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ea4e7a2975a1ebcaa8c2f702cfb9e1226e6d22f82b3adc3d0a93debfcbcaf94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ea4e7a2975a1ebcaa8c2f702cfb9e1226e6d22f82b3adc3d0a93debfcbcaf94/userdata/shm major:0 minor:387 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1805efbbea12a1ecdaa692cc89eca2527e953f2ec24b3ca720aa26f5a83e1d44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1805efbbea12a1ecdaa692cc89eca2527e953f2ec24b3ca720aa26f5a83e1d44/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e080e297c85fb9ba3375b4d44266b365447bbb77d4e4ccab33a1f5ec2eebbce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e080e297c85fb9ba3375b4d44266b365447bbb77d4e4ccab33a1f5ec2eebbce/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36281239ffcf777e5a4afa756d27d386564fea92903f5f5e061ddc15f4e54d63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36281239ffcf777e5a4afa756d27d386564fea92903f5f5e061ddc15f4e54d63/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37fe08941fdd657eecd19192333b5b2988fdb45578c4f864cd87dc1a0e9131cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37fe08941fdd657eecd19192333b5b2988fdb45578c4f864cd87dc1a0e9131cd/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b3fccaca1ac6b70542db96f4dc40b7e60cf554e3717387803f741cbdacb2979/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b3fccaca1ac6b70542db96f4dc40b7e60cf554e3717387803f741cbdacb2979/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e4b2496ed0a5b0e53c97183c8ab28ee80817e1bb6155fa0f24dec6f9697c6b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e4b2496ed0a5b0e53c97183c8ab28ee80817e1bb6155fa0f24dec6f9697c6b8/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4efc45b23dedf56d295be7a90bd7459d2ba8a9a5178117bfe43f127612e0e9c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4efc45b23dedf56d295be7a90bd7459d2ba8a9a5178117bfe43f127612e0e9c8/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62becf816bd512c417ea5154b84545444e20cc11702b8bb28dd56c1cfe3c5647/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62becf816bd512c417ea5154b84545444e20cc11702b8bb28dd56c1cfe3c5647/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77309725d225f5c436ce0c905317d57497ae331ecd2d652d94bd1ce6282e55cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77309725d225f5c436ce0c905317d57497ae331ecd2d652d94bd1ce6282e55cf/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a74c36ea1e9b0a2b54956e883fc150c8225329ecd15713de64ac73848a3dcb06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a74c36ea1e9b0a2b54956e883fc150c8225329ecd15713de64ac73848a3dcb06/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b22032917b559a73c2704d72d879eb3fd30b66666faee15df6737a4515baecd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b22032917b559a73c2704d72d879eb3fd30b66666faee15df6737a4515baecd4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6ebd357cd9ac717b5fac0fcc1dc30fdfefac89ef3c9311a804e41c055e4b6e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6ebd357cd9ac717b5fac0fcc1dc30fdfefac89ef3c9311a804e41c055e4b6e8/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b835c3d5f549f025d891935df7006d3210d293121f7deedc4be6ea290a75b078/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b835c3d5f549f025d891935df7006d3210d293121f7deedc4be6ea290a75b078/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bec7ace172a9d7a6a50e0ebeefca2ecf8acd9775f6e98cfef685653a5c39cf12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bec7ace172a9d7a6a50e0ebeefca2ecf8acd9775f6e98cfef685653a5c39cf12/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5355dddfa638fd8b4c77bbd82acbaeceee8da7e12d2eb5307de664d6dc7dd64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5355dddfa638fd8b4c77bbd82acbaeceee8da7e12d2eb5307de664d6dc7dd64/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb05e997b4603abaaf58dadf4506c57db933d38e4730697385c984223d39fd3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb05e997b4603abaaf58dadf4506c57db933d38e4730697385c984223d39fd3a/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3f87f4d397e00d4abb66aed03b6ce7f6dc7ee1022c8eee4377c18fd89878bc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3f87f4d397e00d4abb66aed03b6ce7f6dc7ee1022c8eee4377c18fd89878bc2/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~projected/kube-api-access-zkmhn:{mountpoint:/var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~projected/kube-api-access-zkmhn major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/128c7b38-4784-403c-be55-e2f1561c2f72/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/128c7b38-4784-403c-be55-e2f1561c2f72/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~projected/kube-api-access-9mbfv:{mountpoint:/var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~projected/kube-api-access-9mbfv major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~projected/kube-api-access-bmc2q:{mountpoint:/var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~projected/kube-api-access-bmc2q major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volumes/kubernetes.io~projected/kube-api-access-mm5r5:{mountpoint:/var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volumes/kubernetes.io~projected/kube-api-access-mm5r5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b359175-b607-449b-94ee-86cd7923be08/volumes/kubernetes.io~projected/kube-api-access-l4vll:{mountpoint:/var/lib/kubelet/pods/3b359175-b607-449b-94ee-86cd7923be08/volumes/kubernetes.io~projected/kube-api-access-l4vll major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54c97174-d65c-4516-923f-c504861b0196/volumes/kubernetes.io~projected/kube-api-access-9r48b:{mountpoint:/var/lib/kubelet/pods/54c97174-d65c-4516-923f-c504861b0196/volumes/kubernetes.io~projected/kube-api-access-9r48b major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54c97174-d65c-4516-923f-c504861b0196/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/54c97174-d65c-4516-923f-c504861b0196/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~projected/kube-api-access-hx9t7:{mountpoint:/var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~projected/kube-api-access-hx9t7 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~projected/kube-api-access-fbrmk:{mountpoint:/var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~projected/kube-api-access-fbrmk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~projected/kube-api-access-fnfdx:{mountpoint:/var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~projected/kube-api-access-fnfdx major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78bd8e89-b28f-4ce4-abaa-dc668b48014b/volumes/kubernetes.io~projected/kube-api-access-fwb8h:{mountpoint:/var/lib/kubelet/pods/78bd8e89-b28f-4ce4-abaa-dc668b48014b/volumes/kubernetes.io~projected/kube-api-access-fwb8h major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66/volumes/kubernetes.io~projected/kube-api-access-j95j2:{mountpoint:/var/lib/kubelet/pods/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66/volumes/kubernetes.io~projected/kube-api-access-j95j2 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c23f988-ab24-48b8-b91d-c56a1d852f5d/volumes/kubernetes.io~projected/kube-api-access-f7cvv:{mountpoint:/var/lib/kubelet/pods/8c23f988-ab24-48b8-b91d-c56a1d852f5d/volumes/kubernetes.io~projected/kube-api-access-f7cvv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d064689-ea69-4221-b860-9242ee3f80a0/volumes/kubernetes.io~projected/kube-api-access-vqk9g:{mountpoint:/var/lib/kubelet/pods/8d064689-ea69-4221-b860-9242ee3f80a0/volumes/kubernetes.io~projected/kube-api-access-vqk9g major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/915c8081-3df3-4ada-adca-74cb14da66de/volumes/kubernetes.io~projected/kube-api-access-q84ml:{mountpoint:/var/lib/kubelet/pods/915c8081-3df3-4ada-adca-74cb14da66de/volumes/kubernetes.io~projected/kube-api-access-q84ml major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7/volumes/kubernetes.io~projected/kube-api-access-56dwj:{mountpoint:/var/lib/kubelet/pods/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7/volumes/kubernetes.io~projected/kube-api-access-56dwj major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7/volumes/kubernetes.io~secret/metrics-tls major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~projected/kube-api-access-r5rt5:{mountpoint:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~projected/kube-api-access-r5rt5 major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b404292b-90b0-42b0-bc52-1fb2b5884250/volumes/kubernetes.io~projected/kube-api-access-x4xrl:{mountpoint:/var/lib/kubelet/pods/b404292b-90b0-42b0-bc52-1fb2b5884250/volumes/kubernetes.io~projected/kube-api-access-x4xrl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be542312-aee2-46b1-aa3d-f1c5e07cd59c/volumes/kubernetes.io~projected/kube-api-access-xbkg2:{mountpoint:/var/lib/kubelet/pods/be542312-aee2-46b1-aa3d-f1c5e07cd59c/volumes/kubernetes.io~projected/kube-api-access-xbkg2 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df6cbdc5-6506-47c1-a44e-6d185d3fc716/volumes/kubernetes.io~projected/kube-api-access-j4gsj:{mountpoint:/var/lib/kubelet/pods/df6cbdc5-6506-47c1-a44e-6d185d3fc716/volumes/kubernetes.io~projected/kube-api-access-j4gsj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df6cbdc5-6506-47c1-a44e-6d185d3fc716/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/df6cbdc5-6506-47c1-a44e-6d185d3fc716/volumes/kubernetes.io~secret/metrics-certs major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~projected/kube-api-access-62227:{mountpoint:/var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~projected/kube-api-access-62227 major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:366 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/33e6a3ade179bcf05821e60417f5254357a879d659752c84ed1507d274564a6b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b713155549993439371731f4c88478a7497bf6bed432e986f567844016ca7492/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/21c4bbe442a06b0174a4703a700f063323678707a2edf8c9fa3ce46e37c861ef/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ac96757ac753d3bbb7c78fbc73a5bbe14f24cc40fad6ef4880dc33a643f57e4c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/790336d0d1eeedf489a9b03eaa4f313e87a873a8d0c7edbd7984af433d71ec2f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/55e9e72a9f7d194c8e0938cf8c208c0a4694c3a07cadf1d1f6e0425e1dfc7960/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3306efeb7003f001009f05583dc032ac628f5b1ba3d73ee233685a6e9a1674d7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9b2367b1d237effb54b6ae81416e1a869b7f301685a042d86d93a22f79a720ce/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/5c21dc81cb8a3bf55b3c3c5257810125123d2695e1db21287f189610a73b53a6/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/2915467229a2ac81d1d25342e37bb5d8c7d30a3d91b84e5c06a8dc8997b67cfb/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/0ae688502bd216cc30f1a05deed651bc3886726a5f3c5237a56032b5ab1fcd6f/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/9d8be3d8c58fbb31cc3131d81e9e2919781443ad873480f406848cbccf940413/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9b14e2cb8a77cf20921304008ef600c906a4b0d57b2187c2d00bc37394974492/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/2561008d84674db9bd86111544c2f95845c23c5031098f96b4060e5d8886096f/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-160:{mountpoint:/var/lib/containers/storage/overlay/e18439fcb2d8b4b8b9ab050e711d8d2309d0bb18775afc88f00d6570f03dbb01/merged major:0 minor:160 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/759456ba0018ef0488012008b23491e8e5ab00d4f22643e386ca62026111c04c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d94914d1e5e60d6704876063e4bb6af9f10a812270d6f326da3621704b1ecb42/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/df0d6bed13145820d40cb74f0f051fcb222a41e9fd8c6f5e09b3bda22e0afde6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a0eeb903e5037d1719c15ae5315bd385cea0793f9c9077c64ba47d48af00e277/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ffe9de9425343e194848ccbbc5222d5ec09ed71ca54fff962ef4df7c63212f61/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/dcda7ea521e9c975ab81e89fa0bdd1b855afe5646f678d4119f7d054842cd393/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/5c600e8dc471c654fbb42c667a0d7d379b7b24254398ddddc2aa915555d9b10c/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/1bc4a95271a9804c0c0c677575b0246c70b1fc67351b209f80e5f1b4d77a9e6e/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/20153985f25c8c925f7de2ffbac909bbcc16516783d38f2fdf6bfde5795fc4ad/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/29fec97ddb282df089f5497ee70c60eefe3d1f470439f591afa1da8ea9ecbd4e/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/9c60c2acc19662b62a41be4dce7c6957301e22683d0ee02a138bb0539b6ce245/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/ffd44304f1cc55c3f9f03d3773f903c09a2fb5d0fb504f556adc8f85539ee3d8/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/85459b2686b4574ddfbb138423b3fc92bfe400d9cfdb6156173be8b2de12be1e/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/e8f0370c713402d6c8b89c19326074786de524da7eef5c61964aadfc4ddca01c/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ed8e835f4be8193457cc96c33fcb62d5af25467c06e6858386016b6b228c1f64/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/730711d2ddaae95b8f21eefdba20cf2a3686dfb17316b4970acaad9eee5b4aa1/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/0891b28c28465ad5c8cf5ce5ad7ec0505fbdf156211b4dc2c84df88ce8877326/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/6cd71cbb6814a6ab86ab5644f71ce7c26e43917bb6423f762183a65f8292bfc0/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/67a83b141e982b5874625c78dfb25101bbe08d2159c3d4a02fa8cdbeca1ab50a/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/5c1e25dd6cabdf03132fc9e486d778fe4fff727b5c84b81e208eae467a209c7d/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/aa1d343889f1d81c654416543092fa03da1421a3a3199fb22a997f80d9bb29d6/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/27adbdfa9bbe1c9a9d206def83c668fdaf46a67799475ca32db465fd158bda42/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/cead6844696adeb660524d9630cc5e3d9cda42bb8e28aad2fa9384c3394287bd/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/caaaacc7844f56e4ceb3c125df79c646bace1bc59afd75096eec5deec7ea7d4b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/3366272ef6a7a98f3b7d3d8ea4968aa6cb1f0776e42ef451a75e228771fcdca6/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a68cc3da9a820869b387c873787a00051e2b37af7f2bf1e040277a2bf1acda1a/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/fa0c787b2dc4a652c62fbcc13373129273f83b08362aea1cecf2d880d1137ab4/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/2fb79bc33600aedade8e105af0643c60880c22cf9188d10fc2b744d58c159b53/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/9649a7d4c6af0968b65ca5c264a1a81e7b3c5488a4e81d4068b0c25dc9d9bb8a/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/1c3477a34be141b8b5e7339f957a30977f59e20be72784c7bc8946da5fa80080/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/6d79b606b1f02f30de74906dbb81a47823b0b5c380cf8a7aeafeba856c00327f/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/336d5d93f38fb83d4847992dd8ab03a4d41c4cfd396da11b52eca83c1c31eb00/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/3be26cf984d84183e63e4039ecde06526b0d3dd826dc597d7a2314705e0bc329/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/e275712b4c44512a0e4d71527b600c7b72e8b586f74594a43615b8e0a347514a/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/95676b5e0091e7a20ee14db8d51eb46d3dfff7f867c084d4b91783aca8cf220b/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/2a6c5fb1cc329eb86973e508916c6b3f1bd05bdb63ba1345a4f7b9c4f22f5450/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/45107cf39bce272ec5ebfbb7c37563304079fdfa8cfc0d3e64e56f4b4a280c76/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/8919910786d11319d9843b10b9a9b70b5bbe9cd26eba9d783f914b51165033d5/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/bcc9d55469046decee96d605c71655b04a99cf1dd7d076c79c17b6e24898b6e7/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/99ec0ad83b109ae8f182ad1347aa004e8ce8f08b09bc053288b0d26fa393f585/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/1927aa8e1f9e3a3579982e548c652a0d841a4bd9af7943ffc603845520a4c009/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/89d2beeccd3793cba19781e8973b1c88fe0eef9392ef956ab901884bc041abc9/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/f01c0b44ef98265e65d785764094a0f1212c9750e8076fc0f3e076a8598b47e5/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/c83b00a78856257a6600da50de151630afd1ca34642cda69b934a3823361ed84/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5c836c5b31b9f78ed9c78e8c338d2f2ff0947a7ec7bd2d2aeaac3bc7052bdcaf/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/81f855f5bd0c4277653448f718e80253951853824bf337c373b726098e25b497/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/ae6dbf9f192994b2f795bd63d450429b44dda087e2ef218b411a32dfa77b5f7a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/674c7f3fe634e11b7206c5a22b0752ae583c57a252deaf73c5b146c2d0f31b15/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/eaa5d58975ec38a64c923499fad15ceb36f5598b665bbb373540ca771be350d4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/d030d385109a198803bbe1f56596cb72bf31ef42aab29065ff85e0f8f0a50010/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2c999a555e611ba6e322f561552446df312248116ea3492a2b6c2c8ba233116f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/91b778e6b25ccf1582bc137a33c95b22c30be743172b1c57e61bbf7b12865bba/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/be89c26ee3bd9f3add4b11278b86790d67f2b7ae45ae4de78ca03110d7a6f509/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/16c89ac5f3c23e4fb269a959e9ea09a9bc4f8f7cd3e701cfa1d9c8f1fb1e6d55/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 03:43:08.470202 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.469840 6735 manager.go:217] Machine: {Timestamp:2026-11-01 03:43:08.469391635 +0000 UTC m=+0.058486254 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3207545 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23408ad730643961cad07b37f22d67 SystemUUID:ec23408a-d730-6439-61ca-d07b37f22d67 BootID:3fdf7361-e8a1-40ce-bb8d-f36c0812d938 Filesystems:[{Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~projected/kube-api-access-bmc2q DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37fe08941fdd657eecd19192333b5b2988fdb45578c4f864cd87dc1a0e9131cd/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b835c3d5f549f025d891935df7006d3210d293121f7deedc4be6ea290a75b078/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77309725d225f5c436ce0c905317d57497ae331ecd2d652d94bd1ce6282e55cf/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be542312-aee2-46b1-aa3d-f1c5e07cd59c/volumes/kubernetes.io~projected/kube-api-access-xbkg2 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~projected/kube-api-access-r5rt5 DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b359175-b607-449b-94ee-86cd7923be08/volumes/kubernetes.io~projected/kube-api-access-l4vll DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:366 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5355dddfa638fd8b4c77bbd82acbaeceee8da7e12d2eb5307de664d6dc7dd64/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8c23f988-ab24-48b8-b91d-c56a1d852f5d/volumes/kubernetes.io~projected/kube-api-access-f7cvv DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1805efbbea12a1ecdaa692cc89eca2527e953f2ec24b3ca720aa26f5a83e1d44/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b3fccaca1ac6b70542db96f4dc40b7e60cf554e3717387803f741cbdacb2979/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b404292b-90b0-42b0-bc52-1fb2b5884250/volumes/kubernetes.io~projected/kube-api-access-x4xrl DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-160 DeviceMajor:0 DeviceMinor:160 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~projected/kube-api-access-fnfdx DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~projected/kube-api-access-hx9t7 DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4efc45b23dedf56d295be7a90bd7459d2ba8a9a5178117bfe43f127612e0e9c8/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df6cbdc5-6506-47c1-a44e-6d185d3fc716/volumes/kubernetes.io~projected/kube-api-access-j4gsj DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df6cbdc5-6506-47c1-a44e-6d185d3fc716/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:378 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78bd8e89-b28f-4ce4-abaa-dc668b48014b/volumes/kubernetes.io~projected/kube-api-access-fwb8h DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6ebd357cd9ac717b5fac0fcc1dc30fdfefac89ef3c9311a804e41c055e4b6e8/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54c97174-d65c-4516-923f-c504861b0196/volumes/kubernetes.io~projected/kube-api-access-9r48b DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf75ba2-0374-460d-af57-81cebb276672/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~projected/kube-api-access-62227 DeviceMajor:0 DeviceMinor:371 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/128c7b38-4784-403c-be55-e2f1561c2f72/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:383 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volumes/kubernetes.io~projected/kube-api-access-mm5r5 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a74c36ea1e9b0a2b54956e883fc150c8225329ecd15713de64ac73848a3dcb06/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e080e297c85fb9ba3375b4d44266b365447bbb77d4e4ccab33a1f5ec2eebbce/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ea4e7a2975a1ebcaa8c2f702cfb9e1226e6d22f82b3adc3d0a93debfcbcaf94/userdata/shm DeviceMajor:0 DeviceMinor:387 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/07e58839-259c-453a-98f1-dac051fb8f9e/volumes/kubernetes.io~projected/kube-api-access-zkmhn DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~projected/kube-api-access-fbrmk DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66/volumes/kubernetes.io~projected/kube-api-access-j95j2 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bec7ace172a9d7a6a50e0ebeefca2ecf8acd9775f6e98cfef685653a5c39cf12/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~projected/kube-api-access-9mbfv DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/915c8081-3df3-4ada-adca-74cb14da66de/volumes/kubernetes.io~projected/kube-api-access-q84ml DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d064689-ea69-4221-b860-9242ee3f80a0/volumes/kubernetes.io~projected/kube-api-access-vqk9g DeviceMajor:0 DeviceMinor:384 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/67e6497d-1fe5-4ad9-89a5-18997c1bc527/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7/volumes/kubernetes.io~projected/kube-api-access-56dwj DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/34608ae4-4158-4e76-88bd-bdb4d1381f87/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b22032917b559a73c2704d72d879eb3fd30b66666faee15df6737a4515baecd4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb05e997b4603abaaf58dadf4506c57db933d38e4730697385c984223d39fd3a/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54c97174-d65c-4516-923f-c504861b0196/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e4b2496ed0a5b0e53c97183c8ab28ee80817e1bb6155fa0f24dec6f9697c6b8/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f1e0e21-7af3-4316-b343-bcc39d1dcb85/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/38b835f0-2c16-4cee-8787-9914b169aff0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3f87f4d397e00d4abb66aed03b6ce7f6dc7ee1022c8eee4377c18fd89878bc2/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06766e8b91d6988aa5a5fd028a4501bf8670733cae6bc10a5489538d61cbb940/userdata/shm DeviceMajor:0 DeviceMinor:389 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0074608fcbb04640a9f12e496d37ecf08fe99646cce29b5ee6dc19098bcef04a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62becf816bd512c417ea5154b84545444e20cc11702b8bb28dd56c1cfe3c5647/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ab862706-1c0c-4324-86d7-59c9b6313672/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36281239ffcf777e5a4afa756d27d386564fea92903f5f5e061ddc15f4e54d63/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06766e8b91d6988 MacAddress:ee:9c:29:a0:cf:36 Speed:10000 Mtu:8901} {Name:0ea4e7a2975a1eb MacAddress:b6:10:ee:c7:f5:a8 Speed:10000 Mtu:8901} {Name:1e080e297c85fb9 MacAddress:26:07:4e:05:ab:11 Speed:10000 Mtu:8901} {Name:36281239ffcf777 MacAddress:92:9a:07:c3:e3:5e Speed:10000 Mtu:8901} {Name:4b3fccaca1ac6b7 MacAddress:42:8e:53:59:12:d6 Speed:10000 Mtu:8901} {Name:4e4b2496ed0a5b0 MacAddress:06:86:c8:7b:89:b0 Speed:10000 Mtu:8901} {Name:51429a0f0905914 MacAddress:32:99:d7:3d:14:51 Speed:10000 Mtu:8901} {Name:62becf816bd512c MacAddress:a6:73:83:55:8b:9b Speed:10000 Mtu:8901} {Name:bec7ace172a9d7a MacAddress:5a:5e:3d:a7:c6:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c3:1e:9e:b1 Speed:0 Mtu:9001} {Name:br-int MacAddress:16:28:2b:12:a1:c3 Speed:0 Mtu:8901} {Name:c5355dddfa638fd MacAddress:76:98:dc:48:3f:74 Speed:10000 Mtu:8901} {Name:cb05e997b4603ab MacAddress:12:8c:6f:ff:2f:66 Speed:10000 Mtu:8901} {Name:d3f87f4d397e00d MacAddress:ee:58:58:31:cc:32 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c3:1e:9e:b1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:a9:a0:06:86:a8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:00:ed:fa:e3:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 03:43:08.470449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470206 6735 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 03:43:08.470449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470249 6735 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 03:43:08.470629 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470602 6735 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 03:43:08.470789 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470630 6735 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-69.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 03:43:08.470879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470792 6735 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 03:43:08.470879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470805 6735 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 03:43:08.470879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470824 6735 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 03:43:08.470879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470851 6735 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 03:43:08.471057 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.470961 6735 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 03:43:08.471203 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.471189 6735 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 03:43:08.471337 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.471326 6735 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 03:43:08.471394 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.471372 6735 kubelet.go:493] "Attempting to sync node with API server" Nov 01 03:43:08.471394 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.471386 6735 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 03:43:08.471481 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.471400 6735 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 03:43:08.471481 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.471413 6735 kubelet.go:405] "Adding apiserver pod source" Nov 01 03:43:08.471481 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.471424 6735 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 03:43:08.472256 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.472213 6735 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 03:43:08.472256 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.472228 6735 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 03:43:08.472770 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.472745 6735 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 03:43:08.476858 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.476837 6735 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 03:43:08.476858 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.476853 6735 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 03:43:08.476964 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.476954 6735 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 03:43:08.477189 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477175 6735 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 03:43:08.477221 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477206 6735 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 03:43:08.477330 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477320 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 03:43:08.477367 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477336 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 03:43:08.477367 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477345 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 03:43:08.477367 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477354 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 03:43:08.477367 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477364 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477372 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477380 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477385 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477393 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477399 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477412 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477422 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477447 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 03:43:08.477475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.477456 6735 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 03:43:08.480938 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.480921 6735 server.go:1263] "Started kubelet" Nov 01 03:43:08.481402 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.481283 6735 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 03:43:08.481402 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.481394 6735 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 03:43:08.481516 ip-10-0-128-69 systemd[1]: Started Kubernetes Kubelet. Nov 01 03:43:08.481663 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.481018 6735 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 03:43:08.481788 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.481699 6735 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 03:43:08.482848 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.482825 6735 server.go:317] "Adding debug handlers to kubelet server" Nov 01 03:43:08.487941 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.487921 6735 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 03:43:08.488106 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.488058 6735 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:43:08.488181 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.488114 6735 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 03:43:08.488181 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.488122 6735 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 03:43:08.488324 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.488308 6735 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 03:43:08.488383 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.488118 6735 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 03:43:08.490968 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.490406 6735 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:36:55 +0000 UTC" deadline="2028-07-19 16:04:56.396107658 +0000 UTC" Nov 01 03:43:08.490968 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.490447 6735 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15036h21m47.905663827s" Nov 01 03:43:08.490968 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.490716 6735 factory.go:55] Registering systemd factory Nov 01 03:43:08.490968 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.490736 6735 factory.go:223] Registration of the systemd container factory successfully Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.490984 6735 factory.go:153] Registering CRI-O factory Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491001 6735 factory.go:223] Registration of the crio container factory successfully Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491075 6735 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491072 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491106 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491136 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b835f0-2c16-4cee-8787-9914b169aff0" volumeName="kubernetes.io/secret/38b835f0-2c16-4cee-8787-9914b169aff0-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491150 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf75ba2-0374-460d-af57-81cebb276672" volumeName="kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491165 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/projected/ab862706-1c0c-4324-86d7-59c9b6313672-kube-api-access-r5rt5" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491176 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491188 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b835f0-2c16-4cee-8787-9914b169aff0" volumeName="kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-script-lib" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491201 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e58839-259c-453a-98f1-dac051fb8f9e" volumeName="kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491216 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="128c7b38-4784-403c-be55-e2f1561c2f72" volumeName="kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret" seLinuxMountContext="" Nov 01 03:43:08.491223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491229 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34608ae4-4158-4e76-88bd-bdb4d1381f87" volumeName="kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491242 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34608ae4-4158-4e76-88bd-bdb4d1381f87" volumeName="kubernetes.io/projected/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-api-access-bmc2q" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491256 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66" volumeName="kubernetes.io/configmap/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491270 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="915c8081-3df3-4ada-adca-74cb14da66de" volumeName="kubernetes.io/projected/915c8081-3df3-4ada-adca-74cb14da66de-kube-api-access-q84ml" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491283 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e58839-259c-453a-98f1-dac051fb8f9e" volumeName="kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491298 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e58839-259c-453a-98f1-dac051fb8f9e" volumeName="kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-metrics-client-ca" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491310 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf75ba2-0374-460d-af57-81cebb276672" volumeName="kubernetes.io/projected/5cf75ba2-0374-460d-af57-81cebb276672-kube-api-access-hx9t7" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491325 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66" volumeName="kubernetes.io/projected/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-api-access-j95j2" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491337 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7" volumeName="kubernetes.io/empty-dir/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-tmp-dir" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491350 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-metrics-client-ca" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491361 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7" volumeName="kubernetes.io/configmap/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-config-volume" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491372 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7" volumeName="kubernetes.io/projected/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-kube-api-access-56dwj" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491386 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b404292b-90b0-42b0-bc52-1fb2b5884250" volumeName="kubernetes.io/projected/b404292b-90b0-42b0-bc52-1fb2b5884250-kube-api-access-x4xrl" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491401 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b835f0-2c16-4cee-8787-9914b169aff0" volumeName="kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-config" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491415 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78bd8e89-b28f-4ce4-abaa-dc668b48014b" volumeName="kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491427 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d064689-ea69-4221-b860-9242ee3f80a0" volumeName="kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491439 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-federate-client-tls" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491452 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-client-tls" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491467 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216" volumeName="kubernetes.io/configmap/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-konnectivity-ca" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491498 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491511 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1e0e21-7af3-4316-b343-bcc39d1dcb85" volumeName="kubernetes.io/projected/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-kube-api-access-fnfdx" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491525 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78bd8e89-b28f-4ce4-abaa-dc668b48014b" volumeName="kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-binary-copy" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491538 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66" volumeName="kubernetes.io/secret/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491549 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491560 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b404292b-90b0-42b0-bc52-1fb2b5884250" volumeName="kubernetes.io/configmap/b404292b-90b0-42b0-bc52-1fb2b5884250-serviceca" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491573 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491586 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491598 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="915c8081-3df3-4ada-adca-74cb14da66de" volumeName="kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-multus-daemon-config" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491612 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34608ae4-4158-4e76-88bd-bdb4d1381f87" volumeName="kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-metrics-client-ca" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491624 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34608ae4-4158-4e76-88bd-bdb4d1381f87" volumeName="kubernetes.io/empty-dir/34608ae4-4158-4e76-88bd-bdb4d1381f87-volume-directive-shadow" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491635 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66" volumeName="kubernetes.io/empty-dir/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-data-volume" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491646 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491659 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert" seLinuxMountContext="" Nov 01 03:43:08.491642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491093 6735 factory.go:103] Registering Raw factory Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491673 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b359175-b607-449b-94ee-86cd7923be08" volumeName="kubernetes.io/projected/3b359175-b607-449b-94ee-86cd7923be08-kube-api-access-l4vll" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491687 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c23f988-ab24-48b8-b91d-c56a1d852f5d" volumeName="kubernetes.io/projected/8c23f988-ab24-48b8-b91d-c56a1d852f5d-kube-api-access-f7cvv" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491696 6735 manager.go:1199] Started watching for new ooms in manager Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491698 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491710 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e58839-259c-453a-98f1-dac051fb8f9e" volumeName="kubernetes.io/empty-dir/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-textfile" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491724 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491741 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54c97174-d65c-4516-923f-c504861b0196" volumeName="kubernetes.io/secret/54c97174-d65c-4516-923f-c504861b0196-cert" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491753 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34608ae4-4158-4e76-88bd-bdb4d1381f87" volumeName="kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-tls" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491844 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf75ba2-0374-460d-af57-81cebb276672" volumeName="kubernetes.io/configmap/5cf75ba2-0374-460d-af57-81cebb276672-metrics-client-ca" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491863 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78bd8e89-b28f-4ce4-abaa-dc668b48014b" volumeName="kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491877 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="915c8081-3df3-4ada-adca-74cb14da66de" volumeName="kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-cni-binary-copy" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491912 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491928 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216" volumeName="kubernetes.io/secret/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-agent-certs" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.491941 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492010 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492058 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf75ba2-0374-460d-af57-81cebb276672" volumeName="kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-tls" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492074 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67e6497d-1fe5-4ad9-89a5-18997c1bc527" volumeName="kubernetes.io/projected/67e6497d-1fe5-4ad9-89a5-18997c1bc527-kube-api-access-fbrmk" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492108 6735 manager.go:319] Starting recovery of all containers Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492087 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78bd8e89-b28f-4ce4-abaa-dc668b48014b" volumeName="kubernetes.io/projected/78bd8e89-b28f-4ce4-abaa-dc668b48014b-kube-api-access-fwb8h" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492424 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e58839-259c-453a-98f1-dac051fb8f9e" volumeName="kubernetes.io/projected/07e58839-259c-453a-98f1-dac051fb8f9e-kube-api-access-zkmhn" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492482 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67e6497d-1fe5-4ad9-89a5-18997c1bc527" volumeName="kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-tuned" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492495 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1e0e21-7af3-4316-b343-bcc39d1dcb85" volumeName="kubernetes.io/configmap/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-metrics-client-ca" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492559 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1e0e21-7af3-4316-b343-bcc39d1dcb85" volumeName="kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492573 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b835f0-2c16-4cee-8787-9914b169aff0" volumeName="kubernetes.io/projected/38b835f0-2c16-4cee-8787-9914b169aff0-kube-api-access-mm5r5" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492607 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df6cbdc5-6506-47c1-a44e-6d185d3fc716" volumeName="kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492630 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492643 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc030034-30dc-4b76-836d-c0516aa54384" volumeName="kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492654 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e58839-259c-453a-98f1-dac051fb8f9e" volumeName="kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-tls" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492705 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b835f0-2c16-4cee-8787-9914b169aff0" volumeName="kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-env-overrides" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492721 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c23f988-ab24-48b8-b91d-c56a1d852f5d" volumeName="kubernetes.io/empty-dir/8c23f988-ab24-48b8-b91d-c56a1d852f5d-tmp-dir" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492736 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492772 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be542312-aee2-46b1-aa3d-f1c5e07cd59c" volumeName="kubernetes.io/projected/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kube-api-access-xbkg2" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492788 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34608ae4-4158-4e76-88bd-bdb4d1381f87" volumeName="kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492800 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492840 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fab48c4-836f-455c-9b44-f374e57f6079" volumeName="kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492854 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54c97174-d65c-4516-923f-c504861b0196" volumeName="kubernetes.io/projected/54c97174-d65c-4516-923f-c504861b0196-kube-api-access-9r48b" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492878 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f1e0e21-7af3-4316-b343-bcc39d1dcb85" volumeName="kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-tls" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492902 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b359175-b607-449b-94ee-86cd7923be08" volumeName="kubernetes.io/configmap/3b359175-b607-449b-94ee-86cd7923be08-iptables-alerter-script" seLinuxMountContext="" Nov 01 03:43:08.492913 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492929 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67e6497d-1fe5-4ad9-89a5-18997c1bc527" volumeName="kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-tmp" seLinuxMountContext="" Nov 01 03:43:08.494205 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492942 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7" volumeName="kubernetes.io/secret/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-metrics-tls" seLinuxMountContext="" Nov 01 03:43:08.494205 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492966 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab862706-1c0c-4324-86d7-59c9b6313672" volumeName="kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 01 03:43:08.494205 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492978 6735 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df6cbdc5-6506-47c1-a44e-6d185d3fc716" volumeName="kubernetes.io/projected/df6cbdc5-6506-47c1-a44e-6d185d3fc716-kube-api-access-j4gsj" seLinuxMountContext="" Nov 01 03:43:08.494205 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492992 6735 reconstruct.go:97] "Volume reconstruction finished" Nov 01 03:43:08.494205 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.492999 6735 reconciler.go:29] "Reconciler: start to sync state" Nov 01 03:43:08.501664 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.501576 6735 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 03:43:08.513194 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.513171 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:43:08.513981 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.513960 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:43:08.517429 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.517404 6735 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 03:43:08.517516 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.517432 6735 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 03:43:08.517516 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.517453 6735 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 03:43:08.517516 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.517490 6735 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 03:43:08.546944 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.546903 6735 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="6dfcf848db2362c3ce7be8ad965079428218e6da8ccee56a75a4f1f87f31fcb9" exitCode=0 Nov 01 03:43:08.546944 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.546945 6735 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="facc551ac70f0bd072c7ba8cb023a96d1e434e0d5c8630ff0a5dec92d44cc1f2" exitCode=0 Nov 01 03:43:08.547145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.546956 6735 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="f5188af0ca035aa1c209e861fda18cf9bebf1c7ef9f675d2ec51e3fe1b160e72" exitCode=0 Nov 01 03:43:08.547145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.546966 6735 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="02047441698c77fc6a5614577257b090bfebdcd58b1c9461372f7777924a309a" exitCode=0 Nov 01 03:43:08.547145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.546975 6735 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="917264426d5c7b7c9f178daf09f643ece124f507b75e971b96124360246f185c" exitCode=0 Nov 01 03:43:08.547145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.546984 6735 generic.go:356] "Generic (PLEG): container finished" podID="78bd8e89-b28f-4ce4-abaa-dc668b48014b" containerID="ab5438851fbf8abaa007558c6df12d83cd594ea5c41d5faf84927d0c1a804ab5" exitCode=0 Nov 01 03:43:08.550732 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.550705 6735 generic.go:356] "Generic (PLEG): container finished" podID="5cc61107e38660d4c05b158edffa7de6" containerID="9019fc5505e4a141bdec96f3c296a2343311f06e0b68bb3b171948c9be2fb79b" exitCode=0 Nov 01 03:43:08.553029 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.553005 6735 generic.go:356] "Generic (PLEG): container finished" podID="07e58839-259c-453a-98f1-dac051fb8f9e" containerID="cf3d914e5f0cc66b100c2118c823bd532046b0171559289ea45423a02c577ed7" exitCode=0 Nov 01 03:43:08.561979 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.561958 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:43:08.562345 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.562323 6735 generic.go:356] "Generic (PLEG): container finished" podID="38b835f0-2c16-4cee-8787-9914b169aff0" containerID="90be2596562c5226bdfa857f98a60a6a72998c1269b3312d16169e64897c5893" exitCode=1 Nov 01 03:43:08.588163 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.588135 6735 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:43:08.594130 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.594112 6735 manager.go:324] Recovery completed Nov 01 03:43:08.612479 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.612410 6735 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:43:08.614382 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.614361 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:43:08.614448 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.614395 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:43:08.614448 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.614406 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:43:08.616704 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616685 6735 cpu_manager.go:226] "Starting" policy="none" Nov 01 03:43:08.616704 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616701 6735 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:43:08.616817 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616701 6735 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 03:43:08.616817 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616801 6735 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 03:43:08.616963 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616948 6735 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 03:43:08.616995 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616964 6735 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 03:43:08.616995 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616981 6735 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 03:43:08.616995 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616990 6735 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 03:43:08.617089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.616998 6735 policy_none.go:50] "Start" Nov 01 03:43:08.617089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.617012 6735 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 03:43:08.617089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.617023 6735 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 03:43:08.617560 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.617543 6735 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 03:43:08.617625 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.617596 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:43:08.617625 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.617621 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:43:08.617695 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.617630 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:43:08.618397 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.618383 6735 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 03:43:08.618441 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.618401 6735 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 03:43:08.618441 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.618407 6735 policy_none.go:44] "Start" Nov 01 03:43:08.624394 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.624358 6735 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 03:43:08.626916 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.626902 6735 manager.go:341] "Starting Device Plugin manager" Nov 01 03:43:08.626995 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.626924 6735 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 03:43:08.626995 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.626940 6735 server.go:87] "Starting device plugin registration server" Nov 01 03:43:08.627224 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.627209 6735 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 03:43:08.627289 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.627224 6735 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 03:43:08.627342 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.627301 6735 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 03:43:08.627415 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.627387 6735 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 03:43:08.627462 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.627423 6735 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 03:43:08.627462 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.627429 6735 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 03:43:08.627567 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.627486 6735 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:43:08.627412518Z","UUID":"3ae8b9f5-4868-4bff-9b19-97db900fc82a","Handler":null,"Name":"","Endpoint":""} Nov 01 03:43:08.629058 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.629023 6735 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 03:43:08.629165 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.629066 6735 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 03:43:08.629837 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.629817 6735 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 03:43:08.629919 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.629859 6735 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-69.us-west-2.compute.internal\" not found" Nov 01 03:43:08.674371 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.674333 6735 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:43:08.674371 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.674353 6735 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:43:08.674536 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.674445 6735 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 03:43:08.675182 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.675167 6735 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:43:08.727754 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.727719 6735 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 03:43:08.729010 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.728994 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 03:43:08.729106 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.729023 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 03:43:08.729106 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.729056 6735 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-69.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 03:43:08.729165 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.729123 6735 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.766989 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.766962 6735 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 03:43:08.817929 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.817884 6735 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal"] Nov 01 03:43:08.818554 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.818516 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"5cc61107e38660d4c05b158edffa7de6","Type":"ContainerStarted","Data":"d8a04d442de6eec38b26e045ba38c2cd2dedac85b2e152395c16dd1b83da0a6d"} Nov 01 03:43:08.818606 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.818572 6735 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.818673 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.818568 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"5cc61107e38660d4c05b158edffa7de6","Type":"ContainerDied","Data":"9019fc5505e4a141bdec96f3c296a2343311f06e0b68bb3b171948c9be2fb79b"} Nov 01 03:43:08.818731 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.818720 6735 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.818814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.818708 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"5cc61107e38660d4c05b158edffa7de6","Type":"ContainerStarted","Data":"d0b5d5beeed2e3fdfc985922402dadef379c844aa2f6a27d040f9d9f0dd3364a"} Nov 01 03:43:08.818881 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.818868 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"63ad88aaffc4eb2dd4273e995e15f57f","Type":"ContainerStarted","Data":"c0af13e51e36c5cd9c6e219c3a7998c37b8e160970209fb57e9e5e0832f72ddb"} Nov 01 03:43:08.818915 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.818889 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" event={"ID":"63ad88aaffc4eb2dd4273e995e15f57f","Type":"ContainerStarted","Data":"249ee1cbe5c3fc13619693e342172d6c48c57177676faf0eeed44d97cb88e0c5"} Nov 01 03:43:08.831872 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.831852 6735 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.831933 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.831913 6735 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.876272 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.876205 6735 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 03:43:08.876272 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.876258 6735 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.876937 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.876921 6735 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 03:43:08.876972 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:08.876958 6735 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.976457 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.976426 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.976457 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.976455 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63ad88aaffc4eb2dd4273e995e15f57f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"63ad88aaffc4eb2dd4273e995e15f57f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:08.976644 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:08.976476 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.077508 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.077467 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63ad88aaffc4eb2dd4273e995e15f57f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"63ad88aaffc4eb2dd4273e995e15f57f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.077508 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.077482 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/63ad88aaffc4eb2dd4273e995e15f57f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"63ad88aaffc4eb2dd4273e995e15f57f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.077696 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.077527 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.077696 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.077543 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.077696 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.077574 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.077696 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.077622 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5cc61107e38660d4c05b158edffa7de6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" (UID: \"5cc61107e38660d4c05b158edffa7de6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.472664 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.472611 6735 apiserver.go:52] "Watching apiserver" Nov 01 03:43:09.569880 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.569851 6735 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 03:43:09.571220 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.571181 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt","kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal","openshift-dns/node-resolver-x7gtf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-x2ld9","openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs","kube-system/global-pull-secret-syncer-nwbf4","openshift-multus/multus-additional-cni-plugins-74gr2","openshift-multus/network-metrics-daemon-h7fxk","openshift-ingress-canary/ingress-canary-fvvjf","openshift-console/console-6bd4ff9647-pldxz","openshift-dns/dns-default-7kp4z","openshift-multus/multus-5g5xz","openshift-image-registry/node-ca-f79cb","openshift-console/console-56d996f7-5ldfz","openshift-monitoring/node-exporter-r4gkc","openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf","openshift-ovn-kubernetes/ovnkube-node-8kr8k","kube-system/konnectivity-agent-k2s25","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757","openshift-network-diagnostics/network-check-target-r4j7n","openshift-network-operator/iptables-alerter-l8dmf","openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8","openshift-insights/insights-runtime-extractor-lzs95"] Nov 01 03:43:09.576889 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.576872 6735 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.576889 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.576878 6735 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.590458 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.590427 6735 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 03:43:09.623740 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.623716 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qv2jj\"" Nov 01 03:43:09.623900 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.623865 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-v2zmn\"" Nov 01 03:43:09.623988 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.623968 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tmt47\"" Nov 01 03:43:09.624065 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.623996 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jzxr2\"" Nov 01 03:43:09.624065 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.623997 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 03:43:09.624160 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.624061 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7tm86\"" Nov 01 03:43:09.624160 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.624133 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 03:43:09.624643 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.624622 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h4w6m\"" Nov 01 03:43:09.624769 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.624728 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.624769 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.624764 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.624949 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.624931 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.625099 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625078 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.625178 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625101 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 03:43:09.625178 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625111 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.625178 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625099 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 03:43:09.625178 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625163 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.625401 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625387 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.625673 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625645 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.625774 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625704 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.625774 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625709 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.625885 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625869 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 03:43:09.625933 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.625903 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zn4h5\"" Nov 01 03:43:09.626140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626094 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jjmfj\"" Nov 01 03:43:09.626140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626105 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-prgzf\"" Nov 01 03:43:09.626140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626105 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 03:43:09.626140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626117 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.626140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626124 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 03:43:09.626140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626137 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 03:43:09.626374 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626149 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 03:43:09.626374 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626165 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-69w79\"" Nov 01 03:43:09.626560 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626543 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 03:43:09.626560 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626544 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.626709 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626572 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.626709 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626599 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.626709 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626677 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 03:43:09.626709 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626678 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 03:43:09.626847 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626735 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v5dfc\"" Nov 01 03:43:09.626847 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.626829 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 03:43:09.627145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.627127 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 03:43:09.633924 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.633911 6735 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 03:43:09.633980 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:09.633950 6735 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.634398 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.634386 6735 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 03:43:09.634436 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:09.634416 6735 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-69.us-west-2.compute.internal" Nov 01 03:43:09.664359 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664333 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.664484 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664423 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.664484 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664469 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 03:43:09.664581 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664520 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.664581 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664535 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 03:43:09.664581 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664562 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 03:43:09.664827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664798 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w8pg8\"" Nov 01 03:43:09.664909 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.664859 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx8x8\"" Nov 01 03:43:09.665501 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.665482 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 03:43:09.665642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.665628 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.665772 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.665759 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.665859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.665847 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.665963 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.665950 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 03:43:09.666117 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666104 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 03:43:09.666231 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666219 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 03:43:09.666326 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666314 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 03:43:09.666497 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666478 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 03:43:09.666600 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666585 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 03:43:09.666709 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666695 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 03:43:09.666938 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666919 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.667009 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.666948 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ztkjl\"" Nov 01 03:43:09.667089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.667013 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.667089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.667024 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 03:43:09.667651 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.667630 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 03:43:09.667750 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.667689 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 03:43:09.667810 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.667790 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 03:43:09.667865 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.667829 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 03:43:09.667923 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.667888 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 03:43:09.668119 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.668103 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dc6ls\"" Nov 01 03:43:09.668855 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.668832 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv2p7\"" Nov 01 03:43:09.668855 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.668834 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8fn5p\"" Nov 01 03:43:09.669019 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.668877 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2mnqj\"" Nov 01 03:43:09.669019 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669012 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 03:43:09.669163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669065 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 03:43:09.669163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669122 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hqffg\"" Nov 01 03:43:09.669163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669126 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v6qjk\"" Nov 01 03:43:09.669163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669129 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 03:43:09.669163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669157 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 03:43:09.669797 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669780 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 03:43:09.669797 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669792 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 03:43:09.669879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669782 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 03:43:09.669879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669812 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 03:43:09.669879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.669826 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zddqk\"" Nov 01 03:43:09.670220 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.670207 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 03:43:09.670289 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.670207 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 03:43:09.670289 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.670213 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 03:43:09.670457 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.670444 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 03:43:09.673501 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.673482 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 03:43:09.673638 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.673622 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 03:43:09.680980 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.680961 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.681081 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.680996 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-log-socket\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.681081 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681020 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-agent-certs\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:43:09.681177 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681141 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-hostroot\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.681177 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681167 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/34608ae4-4158-4e76-88bd-bdb4d1381f87-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.681261 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681186 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-tmp-dir\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.681261 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681207 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-metrics-client-ca\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.681261 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681254 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.681399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681304 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/34608ae4-4158-4e76-88bd-bdb4d1381f87-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.681399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681324 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-tmp-dir\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.681399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681338 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.681544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681392 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.681544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681411 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-metrics-client-ca\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.681544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681417 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-agent-certs\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:43:09.681544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681425 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-registration-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.681544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681458 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681564 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681567 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681601 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-netd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681625 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-sys-fs\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681644 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-host\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681665 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681669 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-multus-certs\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681707 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cvv\" (UniqueName: \"kubernetes.io/projected/8c23f988-ab24-48b8-b91d-c56a1d852f5d-kube-api-access-f7cvv\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:43:09.681767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681757 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-kubernetes\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681869 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681899 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681926 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-binary-copy\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681962 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.681986 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682017 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-data-volume\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682062 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-multus\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682089 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-conf-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682101 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682114 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682151 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-data-volume\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682148 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-lib-modules\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.682196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682194 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cni-binary-copy\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682201 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-bin\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682255 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682271 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b404292b-90b0-42b0-bc52-1fb2b5884250-host\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682293 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-netns\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682334 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682351 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df6cbdc5-6506-47c1-a44e-6d185d3fc716-metrics-certs\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682359 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682407 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q84ml\" (UniqueName: \"kubernetes.io/projected/915c8081-3df3-4ada-adca-74cb14da66de-kube-api-access-q84ml\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682431 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-tmp\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682462 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682492 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682503 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-tmp\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682502 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-dbus\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682552 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-modprobe-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682589 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682624 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysconfig\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682655 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-slash\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682655 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682708 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38b835f0-2c16-4cee-8787-9914b169aff0-ovn-node-metrics-cert\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682740 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682711 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.682776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682788 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682812 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682850 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682876 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682901 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682916 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/38b835f0-2c16-4cee-8787-9914b169aff0-ovn-node-metrics-cert\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682931 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-ovn\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.682955 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683103 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683142 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-crio-socket\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683182 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmc2q\" (UniqueName: \"kubernetes.io/projected/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-api-access-bmc2q\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683220 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683242 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-kubelet-config\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683270 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683272 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683318 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23f988-ab24-48b8-b91d-c56a1d852f5d-tmp-dir\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683390 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683398 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c23f988-ab24-48b8-b91d-c56a1d852f5d-tmp-dir\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683444 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683470 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683446 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683495 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-metrics-tls\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683528 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9t7\" (UniqueName: \"kubernetes.io/projected/5cf75ba2-0374-460d-af57-81cebb276672-kube-api-access-hx9t7\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683580 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683583 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683580 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683617 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-serving-certs-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683636 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683661 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683662 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf75ba2-0374-460d-af57-81cebb276672-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683684 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683704 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j95j2\" (UniqueName: \"kubernetes.io/projected/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-api-access-j95j2\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.683720 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683715 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-metrics-tls\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683746 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-systemd-units\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683791 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab862706-1c0c-4324-86d7-59c9b6313672-serving-certs-ca-bundle\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683794 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56dwj\" (UniqueName: \"kubernetes.io/projected/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-kube-api-access-56dwj\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683819 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf75ba2-0374-460d-af57-81cebb276672-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683882 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-telemeter-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683903 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683934 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.683974 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684002 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684028 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwb8h\" (UniqueName: \"kubernetes.io/projected/78bd8e89-b28f-4ce4-abaa-dc668b48014b-kube-api-access-fwb8h\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684070 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62227\" (UniqueName: \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684088 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684096 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-device-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684122 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4vll\" (UniqueName: \"kubernetes.io/projected/3b359175-b607-449b-94ee-86cd7923be08-kube-api-access-l4vll\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684148 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r48b\" (UniqueName: \"kubernetes.io/projected/54c97174-d65c-4516-923f-c504861b0196-kube-api-access-9r48b\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684174 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbfv\" (UniqueName: \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684204 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-socket-dir-parent\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684226 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/128c7b38-4784-403c-be55-e2f1561c2f72-original-pull-secret\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684241 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkg2\" (UniqueName: \"kubernetes.io/projected/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kube-api-access-xbkg2\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684261 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684266 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-metrics-client-ca\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684271 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684291 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-node-log\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684316 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-script-lib\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684342 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684366 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684386 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-cnibin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684434 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c23f988-ab24-48b8-b91d-c56a1d852f5d-hosts-file\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684463 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684521 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-wtmp\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684553 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cnibin\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684589 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b359175-b607-449b-94ee-86cd7923be08-host-slash\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684612 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-socket-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684626 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684635 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-kubelet\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684637 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e58839-259c-453a-98f1-dac051fb8f9e-metrics-client-ca\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684636 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78bd8e89-b28f-4ce4-abaa-dc668b48014b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684672 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-script-lib\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684684 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-federate-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684708 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684727 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-textfile\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684747 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684757 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-config\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684782 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684814 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-textfile\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684820 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684850 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684912 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-federate-client-tls\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684945 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684956 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-run\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.684989 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-env-overrides\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685025 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-tls\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685047 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-ovnkube-config\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685088 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5r5\" (UniqueName: \"kubernetes.io/projected/38b835f0-2c16-4cee-8787-9914b169aff0-kube-api-access-mm5r5\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685121 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-cni-binary-copy\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685147 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685145 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685165 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-conf\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685146 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/38b835f0-2c16-4cee-8787-9914b169aff0-env-overrides\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685194 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-systemd\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685200 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-tls\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685212 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-sys\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685234 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-var-lib-kubelet\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685251 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrmk\" (UniqueName: \"kubernetes.io/projected/67e6497d-1fe5-4ad9-89a5-18997c1bc527-kube-api-access-fbrmk\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.685267 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685294 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-cni-binary-copy\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685354 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-root\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685385 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685414 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-system-cni-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685438 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-systemd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685458 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685480 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-system-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685498 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-bin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685518 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685531 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-kubelet\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685555 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-multus-daemon-config\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685578 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfdx\" (UniqueName: \"kubernetes.io/projected/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-kube-api-access-fnfdx\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685593 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmhn\" (UniqueName: \"kubernetes.io/projected/07e58839-259c-453a-98f1-dac051fb8f9e-kube-api-access-zkmhn\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685608 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xrl\" (UniqueName: \"kubernetes.io/projected/b404292b-90b0-42b0-bc52-1fb2b5884250-kube-api-access-x4xrl\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685628 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685653 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rt5\" (UniqueName: \"kubernetes.io/projected/ab862706-1c0c-4324-86d7-59c9b6313672-kube-api-access-r5rt5\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685677 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-etc-kubernetes\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685699 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-sys\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685722 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-var-lib-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685735 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/915c8081-3df3-4ada-adca-74cb14da66de-multus-daemon-config\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685747 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b404292b-90b0-42b0-bc52-1fb2b5884250-serviceca\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685781 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-config-volume\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685859 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685894 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685914 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cf75ba2-0374-460d-af57-81cebb276672-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685970 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.685974 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-config-volume\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686023 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-netns\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686026 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab862706-1c0c-4324-86d7-59c9b6313672-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686050 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686077 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-k8s-cni-cncf-io\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686099 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gsj\" (UniqueName: \"kubernetes.io/projected/df6cbdc5-6506-47c1-a44e-6d185d3fc716-kube-api-access-j4gsj\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686129 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-tuned\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686144 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54c97174-d65c-4516-923f-c504861b0196-cert\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686167 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-etc-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686185 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-konnectivity-ca\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686197 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686208 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686334 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686334 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216-konnectivity-ca\") pod \"konnectivity-agent-k2s25\" (UID: \"c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216\") " pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686366 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-tuned\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686377 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b359175-b607-449b-94ee-86cd7923be08-iptables-alerter-script\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686384 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54c97174-d65c-4516-923f-c504861b0196-cert\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686390 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b404292b-90b0-42b0-bc52-1fb2b5884250-serviceca\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:43:09.687478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.686569 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b359175-b607-449b-94ee-86cd7923be08-iptables-alerter-script\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:43:09.737465 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.737385 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cvv\" (UniqueName: \"kubernetes.io/projected/8c23f988-ab24-48b8-b91d-c56a1d852f5d-kube-api-access-f7cvv\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:43:09.737570 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.737535 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk9g\" (UniqueName: \"kubernetes.io/projected/8d064689-ea69-4221-b860-9242ee3f80a0-kube-api-access-vqk9g\") pod \"network-check-target-r4j7n\" (UID: \"8d064689-ea69-4221-b860-9242ee3f80a0\") " pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:43:09.737883 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.737862 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q84ml\" (UniqueName: \"kubernetes.io/projected/915c8081-3df3-4ada-adca-74cb14da66de-kube-api-access-q84ml\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.742139 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.742123 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56dwj\" (UniqueName: \"kubernetes.io/projected/9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7-kube-api-access-56dwj\") pod \"dns-default-7kp4z\" (UID: \"9727d7d4-17a6-4ae0-a3d9-61c3d8c877b7\") " pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:09.742422 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.742402 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9t7\" (UniqueName: \"kubernetes.io/projected/5cf75ba2-0374-460d-af57-81cebb276672-kube-api-access-hx9t7\") pod \"openshift-state-metrics-7bd7489957-szlw8\" (UID: \"5cf75ba2-0374-460d-af57-81cebb276672\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-szlw8" Nov 01 03:43:09.742602 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.742585 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmc2q\" (UniqueName: \"kubernetes.io/projected/34608ae4-4158-4e76-88bd-bdb4d1381f87-kube-api-access-bmc2q\") pod \"kube-state-metrics-5dc8d7d89b-75rhs\" (UID: \"34608ae4-4158-4e76-88bd-bdb4d1381f87\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-75rhs" Nov 01 03:43:09.743408 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.743388 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwb8h\" (UniqueName: \"kubernetes.io/projected/78bd8e89-b28f-4ce4-abaa-dc668b48014b-kube-api-access-fwb8h\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.743495 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.743428 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j95j2\" (UniqueName: \"kubernetes.io/projected/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-kube-api-access-j95j2\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.749643 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.749614 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gsj\" (UniqueName: \"kubernetes.io/projected/df6cbdc5-6506-47c1-a44e-6d185d3fc716-kube-api-access-j4gsj\") pod \"network-metrics-daemon-h7fxk\" (UID: \"df6cbdc5-6506-47c1-a44e-6d185d3fc716\") " pod="openshift-multus/network-metrics-daemon-h7fxk" Nov 01 03:43:09.749845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.749823 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xrl\" (UniqueName: \"kubernetes.io/projected/b404292b-90b0-42b0-bc52-1fb2b5884250-kube-api-access-x4xrl\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:43:09.749933 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.749912 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkg2\" (UniqueName: \"kubernetes.io/projected/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kube-api-access-xbkg2\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.750144 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.750128 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62227\" (UniqueName: \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\") pod \"console-56d996f7-5ldfz\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:09.751341 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.751324 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmhn\" (UniqueName: \"kubernetes.io/projected/07e58839-259c-453a-98f1-dac051fb8f9e-kube-api-access-zkmhn\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.752981 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.752963 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfdx\" (UniqueName: \"kubernetes.io/projected/6f1e0e21-7af3-4316-b343-bcc39d1dcb85-kube-api-access-fnfdx\") pod \"prometheus-operator-9cd89b74c-7rlnf\" (UID: \"6f1e0e21-7af3-4316-b343-bcc39d1dcb85\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7rlnf" Nov 01 03:43:09.755537 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.755127 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrmk\" (UniqueName: \"kubernetes.io/projected/67e6497d-1fe5-4ad9-89a5-18997c1bc527-kube-api-access-fbrmk\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.755537 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.755190 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r48b\" (UniqueName: \"kubernetes.io/projected/54c97174-d65c-4516-923f-c504861b0196-kube-api-access-9r48b\") pod \"ingress-canary-fvvjf\" (UID: \"54c97174-d65c-4516-923f-c504861b0196\") " pod="openshift-ingress-canary/ingress-canary-fvvjf" Nov 01 03:43:09.755537 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.755244 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4vll\" (UniqueName: \"kubernetes.io/projected/3b359175-b607-449b-94ee-86cd7923be08-kube-api-access-l4vll\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:43:09.755537 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.755439 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5r5\" (UniqueName: \"kubernetes.io/projected/38b835f0-2c16-4cee-8787-9914b169aff0-kube-api-access-mm5r5\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.755537 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.755509 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbfv\" (UniqueName: \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\") pod \"console-6bd4ff9647-pldxz\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:09.756901 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.756881 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rt5\" (UniqueName: \"kubernetes.io/projected/ab862706-1c0c-4324-86d7-59c9b6313672-kube-api-access-r5rt5\") pod \"telemeter-client-7456b8b6fc-lzzzt\" (UID: \"ab862706-1c0c-4324-86d7-59c9b6313672\") " pod="openshift-monitoring/telemeter-client-7456b8b6fc-lzzzt" Nov 01 03:43:09.773748 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.773721 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd4ff9647-pldxz"] Nov 01 03:43:09.787619 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787592 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-device-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787623 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-socket-dir-parent\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787642 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-node-log\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787667 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-cnibin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787689 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c23f988-ab24-48b8-b91d-c56a1d852f5d-hosts-file\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787714 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-wtmp\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787721 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-socket-dir-parent\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787737 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cnibin\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787738 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-cnibin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787760 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b359175-b607-449b-94ee-86cd7923be08-host-slash\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:43:09.787777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787724 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-node-log\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787779 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-node-exporter-wtmp\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787780 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-cnibin\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787794 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b359175-b607-449b-94ee-86cd7923be08-host-slash\") pod \"iptables-alerter-l8dmf\" (UID: \"3b359175-b607-449b-94ee-86cd7923be08\") " pod="openshift-network-operator/iptables-alerter-l8dmf" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787787 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-socket-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787721 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-device-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787771 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c23f988-ab24-48b8-b91d-c56a1d852f5d-hosts-file\") pod \"node-resolver-x7gtf\" (UID: \"8c23f988-ab24-48b8-b91d-c56a1d852f5d\") " pod="openshift-dns/node-resolver-x7gtf" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787822 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-socket-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787837 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-kubelet\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787869 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-run\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787891 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-kubelet\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787897 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787929 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-conf\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787939 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-run\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787944 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787958 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-systemd\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787986 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-sys\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.787993 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-systemd\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788009 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-var-lib-kubelet\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788029 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-conf\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788050 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-root\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788083 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-system-cni-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788075 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-root\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788101 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-systemd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788089 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-var-lib-kubelet\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788051 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-sys\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788121 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788126 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78bd8e89-b28f-4ce4-abaa-dc668b48014b-system-cni-dir\") pod \"multus-additional-cni-plugins-74gr2\" (UID: \"78bd8e89-b28f-4ce4-abaa-dc668b48014b\") " pod="openshift-multus/multus-additional-cni-plugins-74gr2" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788134 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-systemd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788145 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-system-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788161 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-bin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.788163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788168 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788174 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-kubelet\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788210 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-system-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788208 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-bin\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788215 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-etc-kubernetes\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788239 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-kubelet\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788239 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-sys\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788269 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-etc-kubernetes\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788283 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-var-lib-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788292 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e58839-259c-453a-98f1-dac051fb8f9e-sys\") pod \"node-exporter-r4gkc\" (UID: \"07e58839-259c-453a-98f1-dac051fb8f9e\") " pod="openshift-monitoring/node-exporter-r4gkc" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788309 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-netns\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788326 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-k8s-cni-cncf-io\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788353 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-etc-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788368 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-var-lib-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788373 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-netns\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788382 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-log-socket\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788387 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-k8s-cni-cncf-io\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788407 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-hostroot\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788412 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-etc-openvswitch\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788417 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-log-socket\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788432 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788455 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-registration-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788476 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788483 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-hostroot\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788500 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-netd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788516 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788525 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-sys-fs\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788533 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-registration-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788541 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-netd\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788533 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-cni-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788547 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-host\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788581 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-sys-fs\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788579 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-multus-certs\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788592 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-host\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788613 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-multus-certs\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788622 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-kubernetes\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788655 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-multus\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788677 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-conf-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788701 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-lib-modules\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788717 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-kubernetes\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788720 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-var-lib-cni-multus\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788725 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-bin\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788727 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-multus-conf-dir\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788750 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b404292b-90b0-42b0-bc52-1fb2b5884250-host\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788772 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-cni-bin\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788804 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-netns\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788820 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-lib-modules\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788823 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b404292b-90b0-42b0-bc52-1fb2b5884250-host\") pod \"node-ca-f79cb\" (UID: \"b404292b-90b0-42b0-bc52-1fb2b5884250\") " pod="openshift-image-registry/node-ca-f79cb" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788867 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/915c8081-3df3-4ada-adca-74cb14da66de-host-run-netns\") pod \"multus-5g5xz\" (UID: \"915c8081-3df3-4ada-adca-74cb14da66de\") " pod="openshift-multus/multus-5g5xz" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788897 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-dbus\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788917 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-modprobe-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788931 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysconfig\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788945 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-slash\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788970 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788996 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-slash\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789003 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysconfig\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.788997 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789017 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-dbus\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789025 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-modprobe-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789048 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789060 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-ovn\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789066 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-etc-selinux\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789086 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-crio-socket\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789089 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-run-ovn\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789118 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789135 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-kubelet-config\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789150 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789158 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66-crio-socket\") pod \"insights-runtime-extractor-lzs95\" (UID: \"7b2cc2f4-e1a1-4da1-9591-ebeb958d4f66\") " pod="openshift-insights/insights-runtime-extractor-lzs95" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789171 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-systemd-units\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789206 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/128c7b38-4784-403c-be55-e2f1561c2f72-kubelet-config\") pod \"global-pull-secret-syncer-nwbf4\" (UID: \"128c7b38-4784-403c-be55-e2f1561c2f72\") " pod="kube-system/global-pull-secret-syncer-nwbf4" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789214 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/67e6497d-1fe5-4ad9-89a5-18997c1bc527-etc-sysctl-d\") pod \"tuned-x2ld9\" (UID: \"67e6497d-1fe5-4ad9-89a5-18997c1bc527\") " pod="openshift-cluster-node-tuning-operator/tuned-x2ld9" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789225 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/38b835f0-2c16-4cee-8787-9914b169aff0-systemd-units\") pod \"ovnkube-node-8kr8k\" (UID: \"38b835f0-2c16-4cee-8787-9914b169aff0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:09.789215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:09.789250 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be542312-aee2-46b1-aa3d-f1c5e07cd59c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48757\" (UID: \"be542312-aee2-46b1-aa3d-f1c5e07cd59c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48757" Nov 01 03:43:11.781105 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:11.781067 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:11.936146 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:11.936117 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7kp4z" Nov 01 03:43:13.072743 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.072711 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-795f6d897b-ddxst"] Nov 01 03:43:13.076416 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.076393 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.108970 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.108937 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-tmp\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.109135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.108988 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-serving-cert\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.109135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.109010 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsh2\" (UniqueName: \"kubernetes.io/projected/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-kube-api-access-zhsh2\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.109135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.109110 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-service-ca\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.109232 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.109198 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-oauth-serving-cert\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.109232 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.109227 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-trusted-ca-bundle\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.109315 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.109300 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-oauth-config\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.109421 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.109409 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-config\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.137518 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.137496 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795f6d897b-ddxst"] Nov 01 03:43:13.210154 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210124 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-tmp\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210316 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210170 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-serving-cert\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210316 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210284 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsh2\" (UniqueName: \"kubernetes.io/projected/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-kube-api-access-zhsh2\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210316 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210305 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-service-ca\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210324 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-oauth-serving-cert\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210358 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-trusted-ca-bundle\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210397 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-oauth-config\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210809 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210427 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-config\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210809 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210639 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-tmp\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.210809 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210772 6735 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 03:43:13.211004 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.210984 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-config\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.211161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.211138 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-oauth-serving-cert\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.211470 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.211450 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-service-ca\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.211563 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.211486 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-trusted-ca-bundle\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.213234 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.213214 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-serving-cert\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.213315 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.213269 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-console-oauth-config\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.268069 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.268029 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsh2\" (UniqueName: \"kubernetes.io/projected/5a5f6eac-e73f-4261-b04c-320f7e5d0e58-kube-api-access-zhsh2\") pod \"console-795f6d897b-ddxst\" (UID: \"5a5f6eac-e73f-4261-b04c-320f7e5d0e58\") " pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.387190 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.387112 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:13.777653 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:13.777610 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795f6d897b-ddxst"] Nov 01 03:43:13.781591 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:43:13.781562 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5f6eac_e73f_4261_b04c_320f7e5d0e58.slice/crio-ea6a12542fa826542484b59534093c263757b347aacc6894f789f094c214c21a WatchSource:0}: Error finding container ea6a12542fa826542484b59534093c263757b347aacc6894f789f094c214c21a: Status 404 returned error can't find the container with id ea6a12542fa826542484b59534093c263757b347aacc6894f789f094c214c21a Nov 01 03:43:14.441678 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:14.441644 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:14.442159 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:14.441889 6735 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 03:43:14.593874 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:14.593843 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795f6d897b-ddxst" event={"ID":"5a5f6eac-e73f-4261-b04c-320f7e5d0e58","Type":"ContainerStarted","Data":"8bc2f3cc64356faa020a254ec5ece25d3581db1dfe31a8e9f442a46d3848216a"} Nov 01 03:43:14.594071 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:14.593895 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795f6d897b-ddxst" event={"ID":"5a5f6eac-e73f-4261-b04c-320f7e5d0e58","Type":"ContainerStarted","Data":"ea6a12542fa826542484b59534093c263757b347aacc6894f789f094c214c21a"} Nov 01 03:43:14.705408 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:14.705282 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-795f6d897b-ddxst" podStartSLOduration=1.705267375 podStartE2EDuration="1.705267375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:43:14.704617835 +0000 UTC m=+6.293712464" watchObservedRunningTime="2026-11-01 03:43:14.705267375 +0000 UTC m=+6.294362001" Nov 01 03:43:15.498646 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:15.498614 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:15.502191 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:15.502169 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:15.510145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:15.510124 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2s25" Nov 01 03:43:15.883771 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:15.883695 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:16.687458 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:16.687421 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:16.692189 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:16.692168 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:18.286289 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:18.286255 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:18.329735 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:18.329708 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:43:20.692936 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:20.692901 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:20.693359 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:20.693069 6735 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 03:43:20.696745 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:20.696725 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-795f6d897b-ddxst" Nov 01 03:43:20.855906 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:20.855879 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d996f7-5ldfz"] Nov 01 03:43:26.189898 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:26.189868 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8kr8k" Nov 01 03:43:32.999447 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:32.999403 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:43:35.599411 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.599348 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bd4ff9647-pldxz" podUID="2fab48c4-836f-455c-9b44-f374e57f6079" containerName="console" containerID="cri-o://5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d" gracePeriod=15 Nov 01 03:43:35.860808 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.860746 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:35.988739 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988704 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.988739 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988740 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.988960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988778 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.988960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988808 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.988960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988841 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\" (UniqueName: \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.988960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988864 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.988960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988884 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.988960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.988941 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\") pod \"2fab48c4-836f-455c-9b44-f374e57f6079\" (UID: \"2fab48c4-836f-455c-9b44-f374e57f6079\") " Nov 01 03:43:35.989289 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.989223 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:43:35.989289 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.989260 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:35.989487 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.989458 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:35.989487 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.989469 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:35.989487 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.989480 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:35.991371 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.991341 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:43:35.991493 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.991468 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:43:35.991493 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:35.991476 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv" pod "2fab48c4-836f-455c-9b44-f374e57f6079" (UID: "2fab48c4-836f-455c-9b44-f374e57f6079"). InnerVolumeSpecName "kube-api-access-9mbfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:43:36.090007 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.089970 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbfv\" (UniqueName: \"kubernetes.io/projected/2fab48c4-836f-455c-9b44-f374e57f6079-kube-api-access-9mbfv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.090007 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.090001 6735 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-console-config\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.090007 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.090011 6735 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-oauth-config\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.090240 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.090019 6735 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-trusted-ca-bundle\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.090240 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.090071 6735 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.090240 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.090082 6735 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fab48c4-836f-455c-9b44-f374e57f6079-oauth-serving-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.090240 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.090092 6735 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fab48c4-836f-455c-9b44-f374e57f6079-console-serving-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.090240 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.090102 6735 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fab48c4-836f-455c-9b44-f374e57f6079-tmp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:36.666333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.666300 6735 generic.go:356] "Generic (PLEG): container finished" podID="2fab48c4-836f-455c-9b44-f374e57f6079" containerID="5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d" exitCode=0 Nov 01 03:43:36.666876 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.666371 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd4ff9647-pldxz" Nov 01 03:43:36.666876 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.666396 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4ff9647-pldxz" event={"ID":"2fab48c4-836f-455c-9b44-f374e57f6079","Type":"ContainerDied","Data":"5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d"} Nov 01 03:43:36.666876 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.666442 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd4ff9647-pldxz" event={"ID":"2fab48c4-836f-455c-9b44-f374e57f6079","Type":"ContainerDied","Data":"1e080e297c85fb9ba3375b4d44266b365447bbb77d4e4ccab33a1f5ec2eebbce"} Nov 01 03:43:36.666876 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.666461 6735 scope.go:122] "RemoveContainer" containerID="5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d" Nov 01 03:43:36.674416 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.674397 6735 scope.go:122] "RemoveContainer" containerID="5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d" Nov 01 03:43:36.674678 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:36.674657 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d\": container with ID starting with 5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d not found: ID does not exist" containerID="5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d" Nov 01 03:43:36.674733 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.674688 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d"} err="failed to get container status \"5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d\": rpc error: code = NotFound desc = could not find container \"5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d\": container with ID starting with 5b0145db3c2156351db0b2f2ef981a33d0b64db104506f5d75fa56bef91aa04d not found: ID does not exist" Nov 01 03:43:36.820419 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.820341 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd4ff9647-pldxz"] Nov 01 03:43:36.866732 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:36.866702 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bd4ff9647-pldxz"] Nov 01 03:43:38.332879 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:38.332853 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r4j7n" Nov 01 03:43:38.522461 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:38.522391 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fab48c4-836f-455c-9b44-f374e57f6079" path="/var/lib/kubelet/pods/2fab48c4-836f-455c-9b44-f374e57f6079/volumes" Nov 01 03:43:45.881887 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:45.881843 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56d996f7-5ldfz" podUID="fc030034-30dc-4b76-836d-c0516aa54384" containerName="console" containerID="cri-o://39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1" gracePeriod=15 Nov 01 03:43:46.139355 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.139300 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:46.268134 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268101 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268295 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268152 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268295 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268192 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268295 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268218 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268295 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268282 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268465 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268321 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268465 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268382 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268465 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268416 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\" (UniqueName: \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\") pod \"fc030034-30dc-4b76-836d-c0516aa54384\" (UID: \"fc030034-30dc-4b76-836d-c0516aa54384\") " Nov 01 03:43:46.268652 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268499 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:43:46.268652 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268540 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:46.268737 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268719 6735 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-oauth-serving-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.268790 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268741 6735 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc030034-30dc-4b76-836d-c0516aa54384-tmp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.268845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268760 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:46.268943 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268914 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:46.269076 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.268956 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:43:46.270719 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.270683 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:43:46.270814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.270784 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "kube-api-access-62227". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:43:46.270863 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.270833 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config" pod "fc030034-30dc-4b76-836d-c0516aa54384" (UID: "fc030034-30dc-4b76-836d-c0516aa54384"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:43:46.369192 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.369146 6735 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-console-config\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.369192 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.369184 6735 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-trusted-ca-bundle\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.369192 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.369194 6735 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-serving-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.369192 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.369202 6735 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc030034-30dc-4b76-836d-c0516aa54384-console-oauth-config\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.369452 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.369211 6735 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc030034-30dc-4b76-836d-c0516aa54384-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.369452 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.369220 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62227\" (UniqueName: \"kubernetes.io/projected/fc030034-30dc-4b76-836d-c0516aa54384-kube-api-access-62227\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:43:46.702501 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.702469 6735 generic.go:356] "Generic (PLEG): container finished" podID="fc030034-30dc-4b76-836d-c0516aa54384" containerID="39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1" exitCode=0 Nov 01 03:43:46.702669 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.702526 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d996f7-5ldfz" event={"ID":"fc030034-30dc-4b76-836d-c0516aa54384","Type":"ContainerDied","Data":"39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1"} Nov 01 03:43:46.702669 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.702537 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d996f7-5ldfz" Nov 01 03:43:46.702669 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.702565 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d996f7-5ldfz" event={"ID":"fc030034-30dc-4b76-836d-c0516aa54384","Type":"ContainerDied","Data":"51429a0f0905914e5a00f449624db8d95e9fc3c7054a252b3832ec526cc345de"} Nov 01 03:43:46.702669 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.702584 6735 scope.go:122] "RemoveContainer" containerID="39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1" Nov 01 03:43:46.710498 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.710481 6735 scope.go:122] "RemoveContainer" containerID="39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1" Nov 01 03:43:46.710760 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:43:46.710738 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1\": container with ID starting with 39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1 not found: ID does not exist" containerID="39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1" Nov 01 03:43:46.710808 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.710769 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1"} err="failed to get container status \"39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1\": rpc error: code = NotFound desc = could not find container \"39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1\": container with ID starting with 39f1250a3a0c2bc2ae8f261594ff5fb52ea43be6567cd6c9d42d0170af8a86c1 not found: ID does not exist" Nov 01 03:43:46.855808 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.855725 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d996f7-5ldfz"] Nov 01 03:43:46.902261 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:46.902231 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56d996f7-5ldfz"] Nov 01 03:43:48.521965 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:43:48.521932 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc030034-30dc-4b76-836d-c0516aa54384" path="/var/lib/kubelet/pods/fc030034-30dc-4b76-836d-c0516aa54384/volumes" Nov 01 03:44:02.998484 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:44:02.998446 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:44:33.000208 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:44:33.000169 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:45:02.999221 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:45:02.999182 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:45:32.999014 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:45:32.998980 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:45:41.137539 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.137507 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn"] Nov 01 03:45:41.138004 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.137815 6735 cpu_manager.go:414] "containerMap: removing container" podUID="fc030034-30dc-4b76-836d-c0516aa54384" containerName="console" Nov 01 03:45:41.138004 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.137827 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc030034-30dc-4b76-836d-c0516aa54384" containerName="console" Nov 01 03:45:41.138004 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.137847 6735 cpu_manager.go:414] "containerMap: removing container" podUID="2fab48c4-836f-455c-9b44-f374e57f6079" containerName="console" Nov 01 03:45:41.138004 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.137852 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fab48c4-836f-455c-9b44-f374e57f6079" containerName="console" Nov 01 03:45:41.138004 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.137894 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc030034-30dc-4b76-836d-c0516aa54384" containerName="console" Nov 01 03:45:41.138004 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.137901 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fab48c4-836f-455c-9b44-f374e57f6079" containerName="console" Nov 01 03:45:41.140916 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.140895 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.194852 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.194809 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4c85t\"" Nov 01 03:45:41.241307 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.241280 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 03:45:41.241459 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.241318 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 03:45:41.241835 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.241818 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn"] Nov 01 03:45:41.261859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.261837 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.261976 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.261874 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj52\" (UniqueName: \"kubernetes.io/projected/e08d8c23-f141-471a-b8c0-74624e3b8f56-kube-api-access-blj52\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.261976 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.261908 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.362369 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.362343 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj52\" (UniqueName: \"kubernetes.io/projected/e08d8c23-f141-471a-b8c0-74624e3b8f56-kube-api-access-blj52\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.362546 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.362394 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.362546 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.362448 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.362813 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.362793 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.362889 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.362865 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.423556 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.423486 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj52\" (UniqueName: \"kubernetes.io/projected/e08d8c23-f141-471a-b8c0-74624e3b8f56-kube-api-access-blj52\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.456432 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.456409 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:45:41.619910 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.619878 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6"] Nov 01 03:45:41.623303 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.623284 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.673396 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.673364 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 03:45:41.673396 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.673390 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 03:45:41.673617 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.673428 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 03:45:41.673617 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.673464 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 03:45:41.673617 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.673503 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 03:45:41.673617 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.673586 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 03:45:41.673763 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.673751 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5gxzd\"" Nov 01 03:45:41.690288 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.690264 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6"] Nov 01 03:45:41.765519 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.765487 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.765695 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.765539 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.765695 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.765572 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.765773 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.765688 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gtt\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-kube-api-access-f6gtt\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.765773 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.765733 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.765773 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.765752 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.765870 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.765848 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.851734 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.851705 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn"] Nov 01 03:45:41.855277 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:45:41.855251 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08d8c23_f141_471a_b8c0_74624e3b8f56.slice/crio-091b127aac4bd6612cc1111290d138de8f9c6f27caf0778fbd8c9e3f6c4c43aa WatchSource:0}: Error finding container 091b127aac4bd6612cc1111290d138de8f9c6f27caf0778fbd8c9e3f6c4c43aa: Status 404 returned error can't find the container with id 091b127aac4bd6612cc1111290d138de8f9c6f27caf0778fbd8c9e3f6c4c43aa Nov 01 03:45:41.857195 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.857171 6735 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 03:45:41.866569 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.866546 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gtt\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-kube-api-access-f6gtt\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.866659 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.866586 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.866659 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.866602 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.866659 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.866622 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.866659 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.866640 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.866859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.866800 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.866859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.866837 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.867388 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.867366 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.868944 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.868925 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.869111 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.869094 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.869336 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.869318 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.869412 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.869381 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.925216 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.925162 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.925390 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.925374 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gtt\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-kube-api-access-f6gtt\") pod \"istiod-openshift-gateway-7876bfcb74-nfts6\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:41.934083 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:41.934067 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:42.073272 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:42.073233 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" event={"ID":"e08d8c23-f141-471a-b8c0-74624e3b8f56","Type":"ContainerStarted","Data":"091b127aac4bd6612cc1111290d138de8f9c6f27caf0778fbd8c9e3f6c4c43aa"} Nov 01 03:45:42.344256 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:42.344220 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6"] Nov 01 03:45:42.348500 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:45:42.348475 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05cc3e3_fad8_4655_8577_eb9e42ca8dd1.slice/crio-201448d240f932b92874d86cec7f03c62ee159463b7396a6bd04cb94cbaf1913 WatchSource:0}: Error finding container 201448d240f932b92874d86cec7f03c62ee159463b7396a6bd04cb94cbaf1913: Status 404 returned error can't find the container with id 201448d240f932b92874d86cec7f03c62ee159463b7396a6bd04cb94cbaf1913 Nov 01 03:45:43.082029 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:43.081988 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" event={"ID":"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1","Type":"ContainerStarted","Data":"201448d240f932b92874d86cec7f03c62ee159463b7396a6bd04cb94cbaf1913"} Nov 01 03:45:46.707946 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:46.707916 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:45:46.708197 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:46.707982 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:45:47.099977 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:47.099938 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" event={"ID":"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1","Type":"ContainerStarted","Data":"0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a"} Nov 01 03:45:47.100209 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:47.100163 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:47.101605 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:47.101576 6735 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-nfts6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 03:45:47.101708 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:47.101623 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 03:45:47.294882 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:47.294829 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" podStartSLOduration=1.9382147939999999 podStartE2EDuration="6.294817103s" totalImagesPullingTime="4.356602309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:47.294438049 +0000 UTC m=+158.883532681" watchObservedRunningTime="2026-11-01 03:45:47.294817103 +0000 UTC m=+158.883911732" Nov 01 03:45:48.110951 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:48.110914 6735 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-nfts6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 03:45:48.111353 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:48.110970 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 03:45:51.111082 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:51.111029 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:45:54.132706 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:54.132676 6735 generic.go:356] "Generic (PLEG): container finished" podID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerID="3fae88cc5704de47b13d4a021bbd1800b1930f181c3959893fb63a7a10df31f1" exitCode=0 Nov 01 03:45:54.133115 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:54.132766 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" event={"ID":"e08d8c23-f141-471a-b8c0-74624e3b8f56","Type":"ContainerDied","Data":"3fae88cc5704de47b13d4a021bbd1800b1930f181c3959893fb63a7a10df31f1"} Nov 01 03:45:59.153703 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:59.153663 6735 generic.go:356] "Generic (PLEG): container finished" podID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerID="00b5fe0b50b3ed22b050d79609409172c5d802770bcafe95bdf131a9d6c20c1f" exitCode=0 Nov 01 03:45:59.154211 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:45:59.153711 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" event={"ID":"e08d8c23-f141-471a-b8c0-74624e3b8f56","Type":"ContainerDied","Data":"00b5fe0b50b3ed22b050d79609409172c5d802770bcafe95bdf131a9d6c20c1f"} Nov 01 03:46:03.001901 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:46:03.001860 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:46:10.196227 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:10.196138 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" event={"ID":"e08d8c23-f141-471a-b8c0-74624e3b8f56","Type":"ContainerStarted","Data":"a940fe1590da9ce846ac0b40eb3b7af97b46c772e3d3292a0c442a3f6af31012"} Nov 01 03:46:10.307170 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:10.307106 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" podStartSLOduration=2.95744986 podStartE2EDuration="29.30709247s" totalImagesPullingTime="26.34964261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:46:10.30667245 +0000 UTC m=+181.895767079" watchObservedRunningTime="2026-11-01 03:46:10.30709247 +0000 UTC m=+181.896187101" Nov 01 03:46:11.202911 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:11.202878 6735 generic.go:356] "Generic (PLEG): container finished" podID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerID="a940fe1590da9ce846ac0b40eb3b7af97b46c772e3d3292a0c442a3f6af31012" exitCode=0 Nov 01 03:46:11.203322 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:11.202951 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" event={"ID":"e08d8c23-f141-471a-b8c0-74624e3b8f56","Type":"ContainerDied","Data":"a940fe1590da9ce846ac0b40eb3b7af97b46c772e3d3292a0c442a3f6af31012"} Nov 01 03:46:12.339607 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.339574 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:46:12.430311 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.430281 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-util\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-util\") pod \"e08d8c23-f141-471a-b8c0-74624e3b8f56\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " Nov 01 03:46:12.430479 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.430370 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-bundle\") pod \"e08d8c23-f141-471a-b8c0-74624e3b8f56\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " Nov 01 03:46:12.430479 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.430390 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08d8c23-f141-471a-b8c0-74624e3b8f56-kube-api-access-blj52\" (UniqueName: \"kubernetes.io/projected/e08d8c23-f141-471a-b8c0-74624e3b8f56-kube-api-access-blj52\") pod \"e08d8c23-f141-471a-b8c0-74624e3b8f56\" (UID: \"e08d8c23-f141-471a-b8c0-74624e3b8f56\") " Nov 01 03:46:12.430958 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.430930 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-bundle" pod "e08d8c23-f141-471a-b8c0-74624e3b8f56" (UID: "e08d8c23-f141-471a-b8c0-74624e3b8f56"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:46:12.432634 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.432609 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08d8c23-f141-471a-b8c0-74624e3b8f56-kube-api-access-blj52" pod "e08d8c23-f141-471a-b8c0-74624e3b8f56" (UID: "e08d8c23-f141-471a-b8c0-74624e3b8f56"). InnerVolumeSpecName "kube-api-access-blj52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:46:12.435505 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.435481 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-util" pod "e08d8c23-f141-471a-b8c0-74624e3b8f56" (UID: "e08d8c23-f141-471a-b8c0-74624e3b8f56"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:46:12.531158 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.531138 6735 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-util\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:46:12.531158 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.531160 6735 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e08d8c23-f141-471a-b8c0-74624e3b8f56-bundle\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:46:12.531301 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:12.531172 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj52\" (UniqueName: \"kubernetes.io/projected/e08d8c23-f141-471a-b8c0-74624e3b8f56-kube-api-access-blj52\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:46:13.210916 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:13.210871 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" event={"ID":"e08d8c23-f141-471a-b8c0-74624e3b8f56","Type":"ContainerDied","Data":"091b127aac4bd6612cc1111290d138de8f9c6f27caf0778fbd8c9e3f6c4c43aa"} Nov 01 03:46:13.210916 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:13.210902 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck77dn" Nov 01 03:46:13.210916 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:13.210920 6735 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091b127aac4bd6612cc1111290d138de8f9c6f27caf0778fbd8c9e3f6c4c43aa" Nov 01 03:46:32.127794 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.127715 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp"] Nov 01 03:46:32.128206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.128018 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerName="extract" Nov 01 03:46:32.128206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.128045 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerName="extract" Nov 01 03:46:32.128206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.128066 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerName="util" Nov 01 03:46:32.128206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.128073 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerName="util" Nov 01 03:46:32.128206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.128097 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerName="pull" Nov 01 03:46:32.128206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.128102 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerName="pull" Nov 01 03:46:32.128206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.128155 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08d8c23-f141-471a-b8c0-74624e3b8f56" containerName="extract" Nov 01 03:46:32.130118 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.130099 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.172571 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.172537 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff4f8245-74a9-424d-a8c2-641784bcec20-tmp\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.172704 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.172584 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.172704 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.172658 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.172778 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.172713 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsxl\" (UniqueName: \"kubernetes.io/projected/ff4f8245-74a9-424d-a8c2-641784bcec20-kube-api-access-dtsxl\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.172778 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.172736 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff4f8245-74a9-424d-a8c2-641784bcec20-manifests\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.172778 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.172766 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.260234 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.260199 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 03:46:32.260399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.260238 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 03:46:32.260399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.260360 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 03:46:32.260399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.260384 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 03:46:32.260563 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.260502 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 03:46:32.260563 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.260544 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-t7bc9\"" Nov 01 03:46:32.273319 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.273293 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff4f8245-74a9-424d-a8c2-641784bcec20-tmp\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.273478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.273332 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.273478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.273356 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.273478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.273378 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsxl\" (UniqueName: \"kubernetes.io/projected/ff4f8245-74a9-424d-a8c2-641784bcec20-kube-api-access-dtsxl\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.273478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.273402 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff4f8245-74a9-424d-a8c2-641784bcec20-manifests\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.273478 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.273453 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.273855 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.273826 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ff4f8245-74a9-424d-a8c2-641784bcec20-manifests\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.275705 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.275686 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff4f8245-74a9-424d-a8c2-641784bcec20-tmp\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.276024 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.276002 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.276390 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.276372 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.276701 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.276684 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ff4f8245-74a9-424d-a8c2-641784bcec20-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.352751 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.352726 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsxl\" (UniqueName: \"kubernetes.io/projected/ff4f8245-74a9-424d-a8c2-641784bcec20-kube-api-access-dtsxl\") pod \"opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp\" (UID: \"ff4f8245-74a9-424d-a8c2-641784bcec20\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.378342 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.378277 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp"] Nov 01 03:46:32.442650 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.442619 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:32.916453 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:32.916421 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp"] Nov 01 03:46:32.920939 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:46:32.920908 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4f8245_74a9_424d_a8c2_641784bcec20.slice/crio-c8ea89d99faefa3c3cdb62581424153bb139f665f559087a19df5e5913d03d81 WatchSource:0}: Error finding container c8ea89d99faefa3c3cdb62581424153bb139f665f559087a19df5e5913d03d81: Status 404 returned error can't find the container with id c8ea89d99faefa3c3cdb62581424153bb139f665f559087a19df5e5913d03d81 Nov 01 03:46:32.999193 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:46:32.999154 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:46:33.275772 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:33.275741 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" event={"ID":"ff4f8245-74a9-424d-a8c2-641784bcec20","Type":"ContainerStarted","Data":"c8ea89d99faefa3c3cdb62581424153bb139f665f559087a19df5e5913d03d81"} Nov 01 03:46:39.298150 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:39.298106 6735 generic.go:356] "Generic (PLEG): container finished" podID="ff4f8245-74a9-424d-a8c2-641784bcec20" containerID="ef8f7fe2e50862872a3139f72639bcf6b9986b1e19a1c5c40c9148bc1651bf3f" exitCode=0 Nov 01 03:46:39.298548 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:39.298189 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" event={"ID":"ff4f8245-74a9-424d-a8c2-641784bcec20","Type":"ContainerDied","Data":"ef8f7fe2e50862872a3139f72639bcf6b9986b1e19a1c5c40c9148bc1651bf3f"} Nov 01 03:46:41.307663 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:41.307629 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" event={"ID":"ff4f8245-74a9-424d-a8c2-641784bcec20","Type":"ContainerStarted","Data":"129193261e3c96a89ec2850fd71b46cdeb49672feb22a04b5e7b4dfc16caf54a"} Nov 01 03:46:41.308152 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:41.307763 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:41.577856 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:41.577759 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" podStartSLOduration=3.352937978 podStartE2EDuration="10.577742905s" totalImagesPullingTime="6.758337911s" totalInitContainerRuntime="466.467016ms" isStatefulPod=false podCreationTimestamp="2026-11-01 03:46:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:46:41.531254376 +0000 UTC m=+213.120349007" watchObservedRunningTime="2026-11-01 03:46:41.577742905 +0000 UTC m=+213.166837534" Nov 01 03:46:52.313344 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:52.313311 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bd4ff4fb8-rxlwp" Nov 01 03:46:56.899364 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:56.899321 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67"] Nov 01 03:46:56.902476 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:56.902459 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:56.972451 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:56.972419 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 03:46:56.999428 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:56.999398 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mfq67\"" Nov 01 03:46:56.999428 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:56.999397 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 03:46:57.000151 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.000133 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67"] Nov 01 03:46:57.081197 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.081165 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmr9g\" (UniqueName: \"kubernetes.io/projected/39cb864a-29bd-4112-92d8-8947eb63b4e5-kube-api-access-rmr9g\") pod \"cert-manager-webhook-5dbbbf75cc-cqc67\" (UID: \"39cb864a-29bd-4112-92d8-8947eb63b4e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:57.081349 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.081206 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39cb864a-29bd-4112-92d8-8947eb63b4e5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cqc67\" (UID: \"39cb864a-29bd-4112-92d8-8947eb63b4e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:57.181676 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.181598 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmr9g\" (UniqueName: \"kubernetes.io/projected/39cb864a-29bd-4112-92d8-8947eb63b4e5-kube-api-access-rmr9g\") pod \"cert-manager-webhook-5dbbbf75cc-cqc67\" (UID: \"39cb864a-29bd-4112-92d8-8947eb63b4e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:57.181676 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.181641 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39cb864a-29bd-4112-92d8-8947eb63b4e5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cqc67\" (UID: \"39cb864a-29bd-4112-92d8-8947eb63b4e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:57.239456 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.239425 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39cb864a-29bd-4112-92d8-8947eb63b4e5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cqc67\" (UID: \"39cb864a-29bd-4112-92d8-8947eb63b4e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:57.239580 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.239538 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmr9g\" (UniqueName: \"kubernetes.io/projected/39cb864a-29bd-4112-92d8-8947eb63b4e5-kube-api-access-rmr9g\") pod \"cert-manager-webhook-5dbbbf75cc-cqc67\" (UID: \"39cb864a-29bd-4112-92d8-8947eb63b4e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:57.524811 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.524782 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:46:57.895565 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:57.895488 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67"] Nov 01 03:46:57.898672 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:46:57.898643 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39cb864a_29bd_4112_92d8_8947eb63b4e5.slice/crio-0fdb4499ebd498c18e1cab938478cd20bf1b40cc27e6c63d8b7d747fe286f204 WatchSource:0}: Error finding container 0fdb4499ebd498c18e1cab938478cd20bf1b40cc27e6c63d8b7d747fe286f204: Status 404 returned error can't find the container with id 0fdb4499ebd498c18e1cab938478cd20bf1b40cc27e6c63d8b7d747fe286f204 Nov 01 03:46:58.372284 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:46:58.372251 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" event={"ID":"39cb864a-29bd-4112-92d8-8947eb63b4e5","Type":"ContainerStarted","Data":"0fdb4499ebd498c18e1cab938478cd20bf1b40cc27e6c63d8b7d747fe286f204"} Nov 01 03:47:03.003013 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:47:03.002971 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:47:03.393349 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:03.393257 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" event={"ID":"39cb864a-29bd-4112-92d8-8947eb63b4e5","Type":"ContainerStarted","Data":"7b4c06d419592401ed70efac5e72cc495a3825b40683534273aade58356d4a18"} Nov 01 03:47:03.393349 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:03.393327 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:47:03.502181 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:03.502119 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" podStartSLOduration=2.321316209 podStartE2EDuration="7.502106414s" totalImagesPullingTime="5.180790205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:03.501703823 +0000 UTC m=+235.090798452" watchObservedRunningTime="2026-11-01 03:47:03.502106414 +0000 UTC m=+235.091201042" Nov 01 03:47:04.427571 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.427537 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f668977d-wjqtd"] Nov 01 03:47:04.431174 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.431147 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.480981 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.480946 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 03:47:04.527988 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.527960 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 03:47:04.527988 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.527971 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f88v8\"" Nov 01 03:47:04.528604 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.528585 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f668977d-wjqtd"] Nov 01 03:47:04.543558 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.543531 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77e7502-9975-4ff7-b0e1-687e12a045f8-tls-certs\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.543709 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.543695 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klq6d\" (UniqueName: \"kubernetes.io/projected/a77e7502-9975-4ff7-b0e1-687e12a045f8-kube-api-access-klq6d\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.543761 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.543749 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a77e7502-9975-4ff7-b0e1-687e12a045f8-tmp\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.644885 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.644849 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klq6d\" (UniqueName: \"kubernetes.io/projected/a77e7502-9975-4ff7-b0e1-687e12a045f8-kube-api-access-klq6d\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.645135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.644915 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a77e7502-9975-4ff7-b0e1-687e12a045f8-tmp\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.645135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.644971 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77e7502-9975-4ff7-b0e1-687e12a045f8-tls-certs\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.647658 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.647629 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a77e7502-9975-4ff7-b0e1-687e12a045f8-tmp\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.647849 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.647822 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77e7502-9975-4ff7-b0e1-687e12a045f8-tls-certs\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.703188 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.703103 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klq6d\" (UniqueName: \"kubernetes.io/projected/a77e7502-9975-4ff7-b0e1-687e12a045f8-kube-api-access-klq6d\") pod \"kube-auth-proxy-5f668977d-wjqtd\" (UID: \"a77e7502-9975-4ff7-b0e1-687e12a045f8\") " pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:04.742947 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:04.742914 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" Nov 01 03:47:05.106592 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:05.106555 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f668977d-wjqtd"] Nov 01 03:47:05.109637 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:47:05.109581 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77e7502_9975_4ff7_b0e1_687e12a045f8.slice/crio-2c8a160df8c5f73126d61aa243a5ce8c45e87920fa2f0b0d2024a30da3eb037f WatchSource:0}: Error finding container 2c8a160df8c5f73126d61aa243a5ce8c45e87920fa2f0b0d2024a30da3eb037f: Status 404 returned error can't find the container with id 2c8a160df8c5f73126d61aa243a5ce8c45e87920fa2f0b0d2024a30da3eb037f Nov 01 03:47:05.401977 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:05.401892 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" event={"ID":"a77e7502-9975-4ff7-b0e1-687e12a045f8","Type":"ContainerStarted","Data":"2c8a160df8c5f73126d61aa243a5ce8c45e87920fa2f0b0d2024a30da3eb037f"} Nov 01 03:47:08.867641 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:08.867605 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 03:47:09.399342 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.399314 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cqc67" Nov 01 03:47:09.419551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.419517 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" event={"ID":"a77e7502-9975-4ff7-b0e1-687e12a045f8","Type":"ContainerStarted","Data":"0de98480efe9be423bc5c6ac46435b534574fa6e1d50aeac5d77ec5380c4aa54"} Nov 01 03:47:09.558377 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.558319 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pfp2l"] Nov 01 03:47:09.561959 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.561936 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:47:09.562225 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.562205 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" Nov 01 03:47:09.565128 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.565096 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.584547 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.584513 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aec50471-69e8-4b0e-807a-3d70e92c408e-config-file\") pod \"limitador-limitador-7c9b87d574-4nvf6\" (UID: \"aec50471-69e8-4b0e-807a-3d70e92c408e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.584654 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.584551 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxvr\" (UniqueName: \"kubernetes.io/projected/1b923976-5c43-41cc-ae21-c7b6c29604ac-kube-api-access-8gxvr\") pod \"authorino-56c6c9ddd5-pfp2l\" (UID: \"1b923976-5c43-41cc-ae21-c7b6c29604ac\") " pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" Nov 01 03:47:09.584654 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.584582 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdbd\" (UniqueName: \"kubernetes.io/projected/aec50471-69e8-4b0e-807a-3d70e92c408e-kube-api-access-sjdbd\") pod \"limitador-limitador-7c9b87d574-4nvf6\" (UID: \"aec50471-69e8-4b0e-807a-3d70e92c408e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.623294 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.623266 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mz5bh\"" Nov 01 03:47:09.623462 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.623334 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 03:47:09.623523 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.623445 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 03:47:09.623523 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.623517 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 03:47:09.623804 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.623762 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z76k2\"" Nov 01 03:47:09.624669 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.624646 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:47:09.625448 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.625401 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pfp2l"] Nov 01 03:47:09.670092 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.669980 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f668977d-wjqtd" podStartSLOduration=1.962397859 podStartE2EDuration="5.669965461s" totalImagesPullingTime="3.707567602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:09.669669667 +0000 UTC m=+241.258764296" watchObservedRunningTime="2026-11-01 03:47:09.669965461 +0000 UTC m=+241.259060086" Nov 01 03:47:09.686151 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.686121 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aec50471-69e8-4b0e-807a-3d70e92c408e-config-file\") pod \"limitador-limitador-7c9b87d574-4nvf6\" (UID: \"aec50471-69e8-4b0e-807a-3d70e92c408e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.686306 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.686177 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxvr\" (UniqueName: \"kubernetes.io/projected/1b923976-5c43-41cc-ae21-c7b6c29604ac-kube-api-access-8gxvr\") pod \"authorino-56c6c9ddd5-pfp2l\" (UID: \"1b923976-5c43-41cc-ae21-c7b6c29604ac\") " pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" Nov 01 03:47:09.686306 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.686220 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdbd\" (UniqueName: \"kubernetes.io/projected/aec50471-69e8-4b0e-807a-3d70e92c408e-kube-api-access-sjdbd\") pod \"limitador-limitador-7c9b87d574-4nvf6\" (UID: \"aec50471-69e8-4b0e-807a-3d70e92c408e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.686773 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.686751 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aec50471-69e8-4b0e-807a-3d70e92c408e-config-file\") pod \"limitador-limitador-7c9b87d574-4nvf6\" (UID: \"aec50471-69e8-4b0e-807a-3d70e92c408e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.747132 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.747097 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxvr\" (UniqueName: \"kubernetes.io/projected/1b923976-5c43-41cc-ae21-c7b6c29604ac-kube-api-access-8gxvr\") pod \"authorino-56c6c9ddd5-pfp2l\" (UID: \"1b923976-5c43-41cc-ae21-c7b6c29604ac\") " pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" Nov 01 03:47:09.747277 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.747097 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdbd\" (UniqueName: \"kubernetes.io/projected/aec50471-69e8-4b0e-807a-3d70e92c408e-kube-api-access-sjdbd\") pod \"limitador-limitador-7c9b87d574-4nvf6\" (UID: \"aec50471-69e8-4b0e-807a-3d70e92c408e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.879266 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.879237 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" Nov 01 03:47:09.883356 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.883339 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:09.896587 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:09.896559 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:47:10.255834 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:10.255799 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:47:10.260544 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:47:10.260502 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec50471_69e8_4b0e_807a_3d70e92c408e.slice/crio-df14fddd1fac11ea5770f7412fadf97e0a5b3920bf3d78f912ec1d17d3c41866 WatchSource:0}: Error finding container df14fddd1fac11ea5770f7412fadf97e0a5b3920bf3d78f912ec1d17d3c41866: Status 404 returned error can't find the container with id df14fddd1fac11ea5770f7412fadf97e0a5b3920bf3d78f912ec1d17d3c41866 Nov 01 03:47:10.299120 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:10.299091 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pfp2l"] Nov 01 03:47:10.302061 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:47:10.302023 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b923976_5c43_41cc_ae21_c7b6c29604ac.slice/crio-903993cf8075794b10416e5e5ddafb610b7a689fa4e900e595f2cecc58f4c286 WatchSource:0}: Error finding container 903993cf8075794b10416e5e5ddafb610b7a689fa4e900e595f2cecc58f4c286: Status 404 returned error can't find the container with id 903993cf8075794b10416e5e5ddafb610b7a689fa4e900e595f2cecc58f4c286 Nov 01 03:47:10.427921 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:10.427890 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" event={"ID":"1b923976-5c43-41cc-ae21-c7b6c29604ac","Type":"ContainerStarted","Data":"903993cf8075794b10416e5e5ddafb610b7a689fa4e900e595f2cecc58f4c286"} Nov 01 03:47:10.429067 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:10.429028 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" event={"ID":"aec50471-69e8-4b0e-807a-3d70e92c408e","Type":"ContainerStarted","Data":"df14fddd1fac11ea5770f7412fadf97e0a5b3920bf3d78f912ec1d17d3c41866"} Nov 01 03:47:14.450331 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:14.450296 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" event={"ID":"1b923976-5c43-41cc-ae21-c7b6c29604ac","Type":"ContainerStarted","Data":"aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8"} Nov 01 03:47:14.452519 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:14.452488 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" event={"ID":"aec50471-69e8-4b0e-807a-3d70e92c408e","Type":"ContainerStarted","Data":"ada9977a981afd1003c1b7dff8050a35cc0787502e86eccd24a0abeff75dc1b1"} Nov 01 03:47:14.452636 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:14.452552 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:14.565164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:14.565109 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" podStartSLOduration=1.743486104 podStartE2EDuration="5.56508887s" totalImagesPullingTime="3.821602766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:14.564565659 +0000 UTC m=+246.153660284" watchObservedRunningTime="2026-11-01 03:47:14.56508887 +0000 UTC m=+246.154183499" Nov 01 03:47:14.682823 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:14.682747 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" podStartSLOduration=2.004592489 podStartE2EDuration="5.682731782s" totalImagesPullingTime="3.678139293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:14.681877733 +0000 UTC m=+246.270972361" watchObservedRunningTime="2026-11-01 03:47:14.682731782 +0000 UTC m=+246.271826411" Nov 01 03:47:17.048207 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.048172 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-w2rfl"] Nov 01 03:47:17.052022 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.051993 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.147528 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.147501 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mqsgz\"" Nov 01 03:47:17.148440 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.148422 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-w2rfl"] Nov 01 03:47:17.153063 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.153027 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01b6d2a1-f942-478b-8abb-8a15cbdece37-bound-sa-token\") pod \"cert-manager-597b95cb84-w2rfl\" (UID: \"01b6d2a1-f942-478b-8abb-8a15cbdece37\") " pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.153167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.153149 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmztm\" (UniqueName: \"kubernetes.io/projected/01b6d2a1-f942-478b-8abb-8a15cbdece37-kube-api-access-tmztm\") pod \"cert-manager-597b95cb84-w2rfl\" (UID: \"01b6d2a1-f942-478b-8abb-8a15cbdece37\") " pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.254394 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.254356 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmztm\" (UniqueName: \"kubernetes.io/projected/01b6d2a1-f942-478b-8abb-8a15cbdece37-kube-api-access-tmztm\") pod \"cert-manager-597b95cb84-w2rfl\" (UID: \"01b6d2a1-f942-478b-8abb-8a15cbdece37\") " pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.254589 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.254430 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01b6d2a1-f942-478b-8abb-8a15cbdece37-bound-sa-token\") pod \"cert-manager-597b95cb84-w2rfl\" (UID: \"01b6d2a1-f942-478b-8abb-8a15cbdece37\") " pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.312129 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.312071 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01b6d2a1-f942-478b-8abb-8a15cbdece37-bound-sa-token\") pod \"cert-manager-597b95cb84-w2rfl\" (UID: \"01b6d2a1-f942-478b-8abb-8a15cbdece37\") " pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.312345 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.312321 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmztm\" (UniqueName: \"kubernetes.io/projected/01b6d2a1-f942-478b-8abb-8a15cbdece37-kube-api-access-tmztm\") pod \"cert-manager-597b95cb84-w2rfl\" (UID: \"01b6d2a1-f942-478b-8abb-8a15cbdece37\") " pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.362502 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.362472 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-w2rfl" Nov 01 03:47:17.724471 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:17.724437 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-w2rfl"] Nov 01 03:47:17.727464 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:47:17.727438 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01b6d2a1_f942_478b_8abb_8a15cbdece37.slice/crio-2a5895cbea8129b1b2ea67db48a2b7c18a4df0e06e3a92da1b3309be5d3e83fd WatchSource:0}: Error finding container 2a5895cbea8129b1b2ea67db48a2b7c18a4df0e06e3a92da1b3309be5d3e83fd: Status 404 returned error can't find the container with id 2a5895cbea8129b1b2ea67db48a2b7c18a4df0e06e3a92da1b3309be5d3e83fd Nov 01 03:47:18.469109 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:18.469069 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-w2rfl" event={"ID":"01b6d2a1-f942-478b-8abb-8a15cbdece37","Type":"ContainerStarted","Data":"0563ed47e48afe79e37404c172e40e19845fd5df34fd9fa3e44f030f3defa68f"} Nov 01 03:47:18.469621 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:18.469121 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-w2rfl" event={"ID":"01b6d2a1-f942-478b-8abb-8a15cbdece37","Type":"ContainerStarted","Data":"2a5895cbea8129b1b2ea67db48a2b7c18a4df0e06e3a92da1b3309be5d3e83fd"} Nov 01 03:47:18.578157 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:18.578096 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-w2rfl" podStartSLOduration=1.578082136 podStartE2EDuration="1.578082136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:18.576777905 +0000 UTC m=+250.165872535" watchObservedRunningTime="2026-11-01 03:47:18.578082136 +0000 UTC m=+250.167176767" Nov 01 03:47:25.457481 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:47:25.457449 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4nvf6" Nov 01 03:47:33.000573 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:47:33.000535 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:48:02.999900 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:48:02.999865 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:48:08.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:08.539325 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:48:08.541571 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:08.541551 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:48:32.999781 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:48:32.999742 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:48:33.614540 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.614510 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6"] Nov 01 03:48:33.617640 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.617624 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.667146 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.667116 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.667301 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.667186 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.667301 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.667256 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.667301 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.667292 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.667445 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.667340 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6jr\" (UniqueName: \"kubernetes.io/projected/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-kube-api-access-9s6jr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.667501 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.667453 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.667501 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.667489 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.720504 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.720472 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6"] Nov 01 03:48:33.767928 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.767897 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.768089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.767940 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.768089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.767961 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.768089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.767978 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.768252 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.768173 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6jr\" (UniqueName: \"kubernetes.io/projected/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-kube-api-access-9s6jr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.768252 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.768237 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.768346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.768274 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.768965 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.768939 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.770430 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.770412 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.770763 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.770738 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.770843 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.770823 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.770881 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.770844 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.825612 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.825588 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.825813 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.825794 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6jr\" (UniqueName: \"kubernetes.io/projected/55dbeac6-535c-4fbc-a6e1-3bf88447cd4e-kube-api-access-9s6jr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8lmz6\" (UID: \"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:33.928531 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:33.928462 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:34.337883 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:34.337853 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6"] Nov 01 03:48:34.342238 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:48:34.342209 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dbeac6_535c_4fbc_a6e1_3bf88447cd4e.slice/crio-b367865dd80ead1ab726ada61291dadaf91c0c27279895dd209b5a26736ede7a WatchSource:0}: Error finding container b367865dd80ead1ab726ada61291dadaf91c0c27279895dd209b5a26736ede7a: Status 404 returned error can't find the container with id b367865dd80ead1ab726ada61291dadaf91c0c27279895dd209b5a26736ede7a Nov 01 03:48:34.344227 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:34.344191 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:48:34.344327 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:34.344270 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:48:34.787153 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:34.787116 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" event={"ID":"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e","Type":"ContainerStarted","Data":"fe2f738536199f79f65b9d94e267542f3c57a1b2e95f29563adb638a34f9d949"} Nov 01 03:48:34.787325 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:34.787165 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" event={"ID":"55dbeac6-535c-4fbc-a6e1-3bf88447cd4e","Type":"ContainerStarted","Data":"b367865dd80ead1ab726ada61291dadaf91c0c27279895dd209b5a26736ede7a"} Nov 01 03:48:34.787325 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:34.787196 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:34.902287 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:34.902190 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" podStartSLOduration=1.902176902 podStartE2EDuration="1.902176902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:48:34.901541832 +0000 UTC m=+326.490636473" watchObservedRunningTime="2026-11-01 03:48:34.902176902 +0000 UTC m=+326.491271510" Nov 01 03:48:35.793996 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:35.793962 6735 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8lmz6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 03:48:35.794418 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:35.794016 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" podUID="55dbeac6-535c-4fbc-a6e1-3bf88447cd4e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 03:48:38.793461 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:38.793429 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8lmz6" Nov 01 03:48:39.001614 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.001579 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6"] Nov 01 03:48:39.002076 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.001980 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerName="discovery" containerID="cri-o://0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a" gracePeriod=30 Nov 01 03:48:39.110817 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.110774 6735 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-nfts6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.17:8080/ready\": dial tcp 10.134.0.17:8080: connect: connection refused" start-of-body= Nov 01 03:48:39.110977 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.110844 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.17:8080/ready\": dial tcp 10.134.0.17:8080: connect: connection refused" Nov 01 03:48:39.243542 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.243516 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:48:39.311960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.311928 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-kubeconfig\") pod \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " Nov 01 03:48:39.312161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.311999 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-ca-configmap\") pod \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " Nov 01 03:48:39.312161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.312027 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-local-certs\") pod \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " Nov 01 03:48:39.312161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.312073 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-dns-cert\") pod \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " Nov 01 03:48:39.312161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.312097 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-kube-api-access-f6gtt\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-kube-api-access-f6gtt\") pod \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " Nov 01 03:48:39.312161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.312142 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-token\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-token\") pod \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " Nov 01 03:48:39.312161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.312159 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-cacerts\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-cacerts\") pod \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\" (UID: \"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1\") " Nov 01 03:48:39.312516 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.312492 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-ca-configmap" pod "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" (UID: "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:48:39.314640 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.314608 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-cacerts" pod "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" (UID: "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:48:39.314767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.314670 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-kube-api-access-f6gtt" pod "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" (UID: "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1"). InnerVolumeSpecName "kube-api-access-f6gtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:48:39.314855 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.314836 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-dns-cert" pod "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" (UID: "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:48:39.314933 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.314916 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-local-certs" pod "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" (UID: "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:48:39.315001 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.314972 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-kubeconfig" pod "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" (UID: "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:48:39.315180 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.315165 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-token" pod "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" (UID: "a05cc3e3-fad8-4655-8577-eb9e42ca8dd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:48:39.413534 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.413502 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:48:39.413534 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.413527 6735 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-cacerts\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:48:39.413534 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.413539 6735 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-kubeconfig\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:48:39.413763 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.413549 6735 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-ca-configmap\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:48:39.413763 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.413559 6735 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-local-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:48:39.413763 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.413567 6735 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-istio-csr-dns-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:48:39.413763 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.413576 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gtt\" (UniqueName: \"kubernetes.io/projected/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1-kube-api-access-f6gtt\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:48:39.809466 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.809430 6735 generic.go:356] "Generic (PLEG): container finished" podID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerID="0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a" exitCode=0 Nov 01 03:48:39.809466 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.809470 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" event={"ID":"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1","Type":"ContainerDied","Data":"0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a"} Nov 01 03:48:39.809924 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.809475 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" Nov 01 03:48:39.809924 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.809508 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6" event={"ID":"a05cc3e3-fad8-4655-8577-eb9e42ca8dd1","Type":"ContainerDied","Data":"201448d240f932b92874d86cec7f03c62ee159463b7396a6bd04cb94cbaf1913"} Nov 01 03:48:39.809924 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.809528 6735 scope.go:122] "RemoveContainer" containerID="0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a" Nov 01 03:48:39.818252 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.818231 6735 scope.go:122] "RemoveContainer" containerID="0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a" Nov 01 03:48:39.818514 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:48:39.818496 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a\": container with ID starting with 0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a not found: ID does not exist" containerID="0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a" Nov 01 03:48:39.818554 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.818524 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a"} err="failed to get container status \"0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a\": rpc error: code = NotFound desc = could not find container \"0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a\": container with ID starting with 0967ba5521111b52759495b0929fa84e881089be3ad80f975bc396862bd1b96a not found: ID does not exist" Nov 01 03:48:39.964515 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:39.964480 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6"] Nov 01 03:48:40.010657 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:40.010629 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nfts6"] Nov 01 03:48:40.522880 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:48:40.522845 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" path="/var/lib/kubelet/pods/a05cc3e3-fad8-4655-8577-eb9e42ca8dd1/volumes" Nov 01 03:49:02.999535 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:49:02.999486 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:49:04.859271 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:04.859238 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:49:14.271533 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.271492 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pwkw5"] Nov 01 03:49:14.271969 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.271857 6735 cpu_manager.go:414] "containerMap: removing container" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerName="discovery" Nov 01 03:49:14.271969 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.271873 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerName="discovery" Nov 01 03:49:14.271969 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.271930 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05cc3e3-fad8-4655-8577-eb9e42ca8dd1" containerName="discovery" Nov 01 03:49:14.276072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.276053 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" Nov 01 03:49:14.307732 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.307704 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nhd\" (UniqueName: \"kubernetes.io/projected/76c772d5-99dc-4273-be39-69123b14d696-kube-api-access-t9nhd\") pod \"authorino-6f6944bcdd-pwkw5\" (UID: \"76c772d5-99dc-4273-be39-69123b14d696\") " pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" Nov 01 03:49:14.328915 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.328889 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pwkw5"] Nov 01 03:49:14.408688 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.408656 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nhd\" (UniqueName: \"kubernetes.io/projected/76c772d5-99dc-4273-be39-69123b14d696-kube-api-access-t9nhd\") pod \"authorino-6f6944bcdd-pwkw5\" (UID: \"76c772d5-99dc-4273-be39-69123b14d696\") " pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" Nov 01 03:49:14.445928 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.445897 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth"] Nov 01 03:49:14.449343 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.449320 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.466357 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.466331 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nhd\" (UniqueName: \"kubernetes.io/projected/76c772d5-99dc-4273-be39-69123b14d696-kube-api-access-t9nhd\") pod \"authorino-6f6944bcdd-pwkw5\" (UID: \"76c772d5-99dc-4273-be39-69123b14d696\") " pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" Nov 01 03:49:14.509161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.509134 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcb2e9a-1416-4711-b3af-94b60d002b15-openshift-service-ca\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.509276 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.509167 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcb2e9a-1416-4711-b3af-94b60d002b15-maas-api-tls\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.509317 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.509267 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcb2e9a-1416-4711-b3af-94b60d002b15-metrics-certs\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.509351 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.509331 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvg6n\" (UniqueName: \"kubernetes.io/projected/8bcb2e9a-1416-4711-b3af-94b60d002b15-kube-api-access-lvg6n\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.510763 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.510740 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 03:49:14.510882 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.510779 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 03:49:14.544836 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.544773 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 03:49:14.544836 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.544776 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 03:49:14.544836 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.544792 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qxpmx\"" Nov 01 03:49:14.545824 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.545804 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth"] Nov 01 03:49:14.587739 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.587714 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" Nov 01 03:49:14.609926 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.609899 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvg6n\" (UniqueName: \"kubernetes.io/projected/8bcb2e9a-1416-4711-b3af-94b60d002b15-kube-api-access-lvg6n\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.610067 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.609978 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcb2e9a-1416-4711-b3af-94b60d002b15-openshift-service-ca\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.610067 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.610022 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcb2e9a-1416-4711-b3af-94b60d002b15-maas-api-tls\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.610168 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.610080 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcb2e9a-1416-4711-b3af-94b60d002b15-metrics-certs\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.610705 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.610683 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcb2e9a-1416-4711-b3af-94b60d002b15-openshift-service-ca\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.612556 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.612534 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcb2e9a-1416-4711-b3af-94b60d002b15-metrics-certs\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.612660 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.612572 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcb2e9a-1416-4711-b3af-94b60d002b15-maas-api-tls\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.620270 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.620248 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pwkw5"] Nov 01 03:49:14.667994 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.667960 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvg6n\" (UniqueName: \"kubernetes.io/projected/8bcb2e9a-1416-4711-b3af-94b60d002b15-kube-api-access-lvg6n\") pod \"maas-api-77c84bfd5-s2mth\" (UID: \"8bcb2e9a-1416-4711-b3af-94b60d002b15\") " pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.763934 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.763904 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:14.950987 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:14.950823 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pwkw5"] Nov 01 03:49:14.953913 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:49:14.953883 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76c772d5_99dc_4273_be39_69123b14d696.slice/crio-2002ee34b708cd007a49e879fbda25c4533ffaaf4dfce0f299a9534ec8b07db1 WatchSource:0}: Error finding container 2002ee34b708cd007a49e879fbda25c4533ffaaf4dfce0f299a9534ec8b07db1: Status 404 returned error can't find the container with id 2002ee34b708cd007a49e879fbda25c4533ffaaf4dfce0f299a9534ec8b07db1 Nov 01 03:49:15.128925 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:15.128846 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth"] Nov 01 03:49:15.131831 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:49:15.131803 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcb2e9a_1416_4711_b3af_94b60d002b15.slice/crio-c131b74803aa7cb7e0aae2cc8cda41b36086daf41a0ea3a7513cb8f14d0d7f7d WatchSource:0}: Error finding container c131b74803aa7cb7e0aae2cc8cda41b36086daf41a0ea3a7513cb8f14d0d7f7d: Status 404 returned error can't find the container with id c131b74803aa7cb7e0aae2cc8cda41b36086daf41a0ea3a7513cb8f14d0d7f7d Nov 01 03:49:15.959992 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:15.959951 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" event={"ID":"8bcb2e9a-1416-4711-b3af-94b60d002b15","Type":"ContainerStarted","Data":"c131b74803aa7cb7e0aae2cc8cda41b36086daf41a0ea3a7513cb8f14d0d7f7d"} Nov 01 03:49:15.961167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:15.961145 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" event={"ID":"76c772d5-99dc-4273-be39-69123b14d696","Type":"ContainerStarted","Data":"2002ee34b708cd007a49e879fbda25c4533ffaaf4dfce0f299a9534ec8b07db1"} Nov 01 03:49:16.969375 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:16.968960 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" event={"ID":"76c772d5-99dc-4273-be39-69123b14d696","Type":"ContainerStarted","Data":"721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d"} Nov 01 03:49:16.969375 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:16.969125 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" podUID="76c772d5-99dc-4273-be39-69123b14d696" containerName="authorino" containerID="cri-o://721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d" gracePeriod=30 Nov 01 03:49:17.076687 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.076000 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" podStartSLOduration=1.169067415 podStartE2EDuration="3.075969972s" totalImagesPullingTime="1.906902557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:49:17.075687295 +0000 UTC m=+368.664781926" watchObservedRunningTime="2026-11-01 03:49:17.075969972 +0000 UTC m=+368.665064599" Nov 01 03:49:17.253167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.253136 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" Nov 01 03:49:17.333594 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.333561 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c772d5-99dc-4273-be39-69123b14d696-kube-api-access-t9nhd\" (UniqueName: \"kubernetes.io/projected/76c772d5-99dc-4273-be39-69123b14d696-kube-api-access-t9nhd\") pod \"76c772d5-99dc-4273-be39-69123b14d696\" (UID: \"76c772d5-99dc-4273-be39-69123b14d696\") " Nov 01 03:49:17.335972 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.335941 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c772d5-99dc-4273-be39-69123b14d696-kube-api-access-t9nhd" pod "76c772d5-99dc-4273-be39-69123b14d696" (UID: "76c772d5-99dc-4273-be39-69123b14d696"). InnerVolumeSpecName "kube-api-access-t9nhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:49:17.434406 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.434338 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9nhd\" (UniqueName: \"kubernetes.io/projected/76c772d5-99dc-4273-be39-69123b14d696-kube-api-access-t9nhd\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:49:17.975544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.975512 6735 generic.go:356] "Generic (PLEG): container finished" podID="76c772d5-99dc-4273-be39-69123b14d696" containerID="721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d" exitCode=2 Nov 01 03:49:17.976011 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.975606 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" event={"ID":"76c772d5-99dc-4273-be39-69123b14d696","Type":"ContainerDied","Data":"721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d"} Nov 01 03:49:17.976011 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.975641 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" event={"ID":"76c772d5-99dc-4273-be39-69123b14d696","Type":"ContainerDied","Data":"2002ee34b708cd007a49e879fbda25c4533ffaaf4dfce0f299a9534ec8b07db1"} Nov 01 03:49:17.976011 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.975664 6735 scope.go:122] "RemoveContainer" containerID="721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d" Nov 01 03:49:17.976011 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.975802 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pwkw5" Nov 01 03:49:17.987137 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.987118 6735 scope.go:122] "RemoveContainer" containerID="721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d" Nov 01 03:49:17.987423 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:49:17.987403 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d\": container with ID starting with 721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d not found: ID does not exist" containerID="721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d" Nov 01 03:49:17.987485 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:17.987438 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d"} err="failed to get container status \"721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d\": rpc error: code = NotFound desc = could not find container \"721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d\": container with ID starting with 721596bfbff4f28fed5033c8346d11c73f504245422a384b1a7b49c6e675273d not found: ID does not exist" Nov 01 03:49:18.130432 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:18.130398 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pwkw5"] Nov 01 03:49:18.134113 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:18.134086 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pwkw5"] Nov 01 03:49:18.523831 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:18.523751 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c772d5-99dc-4273-be39-69123b14d696" path="/var/lib/kubelet/pods/76c772d5-99dc-4273-be39-69123b14d696/volumes" Nov 01 03:49:18.982352 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:18.982313 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" event={"ID":"8bcb2e9a-1416-4711-b3af-94b60d002b15","Type":"ContainerStarted","Data":"339c0a767a54139e5947c656c781d8b524bfa89f7a14b9495d423dd9a89d0204"} Nov 01 03:49:18.982859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:18.982387 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:18.988524 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:18.988497 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pfp2l"] Nov 01 03:49:18.988671 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:18.988657 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" podUID="1b923976-5c43-41cc-ae21-c7b6c29604ac" containerName="authorino" containerID="cri-o://aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8" gracePeriod=30 Nov 01 03:49:19.096666 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.096612 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" podStartSLOduration=2.047573861 podStartE2EDuration="5.096597394s" totalImagesPullingTime="3.049023533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:49:19.096062537 +0000 UTC m=+370.685157168" watchObservedRunningTime="2026-11-01 03:49:19.096597394 +0000 UTC m=+370.685692022" Nov 01 03:49:19.230334 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.230308 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" Nov 01 03:49:19.252467 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.252418 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b923976-5c43-41cc-ae21-c7b6c29604ac-kube-api-access-8gxvr\" (UniqueName: \"kubernetes.io/projected/1b923976-5c43-41cc-ae21-c7b6c29604ac-kube-api-access-8gxvr\") pod \"1b923976-5c43-41cc-ae21-c7b6c29604ac\" (UID: \"1b923976-5c43-41cc-ae21-c7b6c29604ac\") " Nov 01 03:49:19.254524 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.254499 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b923976-5c43-41cc-ae21-c7b6c29604ac-kube-api-access-8gxvr" pod "1b923976-5c43-41cc-ae21-c7b6c29604ac" (UID: "1b923976-5c43-41cc-ae21-c7b6c29604ac"). InnerVolumeSpecName "kube-api-access-8gxvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:49:19.353007 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.352972 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gxvr\" (UniqueName: \"kubernetes.io/projected/1b923976-5c43-41cc-ae21-c7b6c29604ac-kube-api-access-8gxvr\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:49:19.987977 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.987946 6735 generic.go:356] "Generic (PLEG): container finished" podID="1b923976-5c43-41cc-ae21-c7b6c29604ac" containerID="aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8" exitCode=0 Nov 01 03:49:19.988551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.987994 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" Nov 01 03:49:19.988551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.988063 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" event={"ID":"1b923976-5c43-41cc-ae21-c7b6c29604ac","Type":"ContainerDied","Data":"aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8"} Nov 01 03:49:19.988551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.988104 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pfp2l" event={"ID":"1b923976-5c43-41cc-ae21-c7b6c29604ac","Type":"ContainerDied","Data":"903993cf8075794b10416e5e5ddafb610b7a689fa4e900e595f2cecc58f4c286"} Nov 01 03:49:19.988551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.988120 6735 scope.go:122] "RemoveContainer" containerID="aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8" Nov 01 03:49:19.996424 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.996403 6735 scope.go:122] "RemoveContainer" containerID="aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8" Nov 01 03:49:19.996715 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:49:19.996698 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8\": container with ID starting with aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8 not found: ID does not exist" containerID="aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8" Nov 01 03:49:19.996758 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:19.996725 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8"} err="failed to get container status \"aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8\": rpc error: code = NotFound desc = could not find container \"aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8\": container with ID starting with aaf5a90758f8d20f1dec8db14cb96264b32d55c1d9016b72fcabab5e905c9bc8 not found: ID does not exist" Nov 01 03:49:20.139788 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:20.139754 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pfp2l"] Nov 01 03:49:20.145844 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:20.145817 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pfp2l"] Nov 01 03:49:20.522842 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:20.522813 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b923976-5c43-41cc-ae21-c7b6c29604ac" path="/var/lib/kubelet/pods/1b923976-5c43-41cc-ae21-c7b6c29604ac/volumes" Nov 01 03:49:24.993048 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:49:24.993008 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77c84bfd5-s2mth" Nov 01 03:49:32.999778 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:49:32.999737 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:50:02.999882 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:50:02.999847 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:50:33.000137 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:50:33.000100 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:50:34.873157 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.873104 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-96g5j"] Nov 01 03:50:34.873582 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.873454 6735 cpu_manager.go:414] "containerMap: removing container" podUID="76c772d5-99dc-4273-be39-69123b14d696" containerName="authorino" Nov 01 03:50:34.873582 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.873467 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c772d5-99dc-4273-be39-69123b14d696" containerName="authorino" Nov 01 03:50:34.873582 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.873482 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1b923976-5c43-41cc-ae21-c7b6c29604ac" containerName="authorino" Nov 01 03:50:34.873582 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.873489 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b923976-5c43-41cc-ae21-c7b6c29604ac" containerName="authorino" Nov 01 03:50:34.873582 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.873557 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c772d5-99dc-4273-be39-69123b14d696" containerName="authorino" Nov 01 03:50:34.873582 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.873565 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b923976-5c43-41cc-ae21-c7b6c29604ac" containerName="authorino" Nov 01 03:50:34.876628 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.876606 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:34.925755 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.925725 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 03:50:34.925755 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.925725 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-6m5fl\"" Nov 01 03:50:34.972143 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.972082 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 03:50:34.972143 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.972082 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 03:50:34.972143 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.972142 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 03:50:34.972908 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.972888 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-96g5j"] Nov 01 03:50:34.982585 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.982560 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtvg\" (UniqueName: \"kubernetes.io/projected/1dfc828d-042d-45ee-b7f1-9da4029c0d41-kube-api-access-tvtvg\") pod \"ai-gateway-controller-7d85675f98-96g5j\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:34.982680 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:34.982592 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfc828d-042d-45ee-b7f1-9da4029c0d41-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-96g5j\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:35.083623 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:35.083590 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtvg\" (UniqueName: \"kubernetes.io/projected/1dfc828d-042d-45ee-b7f1-9da4029c0d41-kube-api-access-tvtvg\") pod \"ai-gateway-controller-7d85675f98-96g5j\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:35.083796 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:35.083629 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfc828d-042d-45ee-b7f1-9da4029c0d41-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-96g5j\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:35.086206 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:35.086183 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfc828d-042d-45ee-b7f1-9da4029c0d41-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-96g5j\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:35.141443 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:35.141371 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtvg\" (UniqueName: \"kubernetes.io/projected/1dfc828d-042d-45ee-b7f1-9da4029c0d41-kube-api-access-tvtvg\") pod \"ai-gateway-controller-7d85675f98-96g5j\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:35.189096 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:35.189060 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:35.549845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:35.549744 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-96g5j"] Nov 01 03:50:35.553192 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:50:35.553163 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfc828d_042d_45ee_b7f1_9da4029c0d41.slice/crio-e2d8180a2137454d8cf2bea540e740eeeca3f73c4bbd25ab3d3f9901851b2f9e WatchSource:0}: Error finding container e2d8180a2137454d8cf2bea540e740eeeca3f73c4bbd25ab3d3f9901851b2f9e: Status 404 returned error can't find the container with id e2d8180a2137454d8cf2bea540e740eeeca3f73c4bbd25ab3d3f9901851b2f9e Nov 01 03:50:36.290916 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:36.290877 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" event={"ID":"1dfc828d-042d-45ee-b7f1-9da4029c0d41","Type":"ContainerStarted","Data":"e2d8180a2137454d8cf2bea540e740eeeca3f73c4bbd25ab3d3f9901851b2f9e"} Nov 01 03:50:39.305457 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:39.305417 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" event={"ID":"1dfc828d-042d-45ee-b7f1-9da4029c0d41","Type":"ContainerStarted","Data":"97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e"} Nov 01 03:50:39.305946 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:39.305487 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:39.416075 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:39.415999 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" podStartSLOduration=2.396028767 podStartE2EDuration="5.415980074s" totalImagesPullingTime="3.019951307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:50:39.415789708 +0000 UTC m=+451.004884338" watchObservedRunningTime="2026-11-01 03:50:39.415980074 +0000 UTC m=+451.005074715" Nov 01 03:50:42.765526 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.765491 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx"] Nov 01 03:50:42.769085 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.769061 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:42.851473 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.851438 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715e8b4-ae01-4678-a074-0409f721d0d0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sdhwx\" (UID: \"7715e8b4-ae01-4678-a074-0409f721d0d0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:42.851646 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.851509 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xsq\" (UniqueName: \"kubernetes.io/projected/7715e8b4-ae01-4678-a074-0409f721d0d0-kube-api-access-g9xsq\") pod \"payload-pre-processing-7b98bb678c-sdhwx\" (UID: \"7715e8b4-ae01-4678-a074-0409f721d0d0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:42.864356 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.864332 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 03:50:42.864634 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.864616 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kv9tx\"" Nov 01 03:50:42.865452 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.865433 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx"] Nov 01 03:50:42.925618 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.925591 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-4qdkd"] Nov 01 03:50:42.928933 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.928914 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:42.952304 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.952282 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715e8b4-ae01-4678-a074-0409f721d0d0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sdhwx\" (UID: \"7715e8b4-ae01-4678-a074-0409f721d0d0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:42.952403 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.952328 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xsq\" (UniqueName: \"kubernetes.io/projected/7715e8b4-ae01-4678-a074-0409f721d0d0-kube-api-access-g9xsq\") pod \"payload-pre-processing-7b98bb678c-sdhwx\" (UID: \"7715e8b4-ae01-4678-a074-0409f721d0d0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:42.952897 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.952878 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7715e8b4-ae01-4678-a074-0409f721d0d0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sdhwx\" (UID: \"7715e8b4-ae01-4678-a074-0409f721d0d0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:42.984998 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:42.984968 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-4qdkd"] Nov 01 03:50:43.010234 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.010210 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xsq\" (UniqueName: \"kubernetes.io/projected/7715e8b4-ae01-4678-a074-0409f721d0d0-kube-api-access-g9xsq\") pod \"payload-pre-processing-7b98bb678c-sdhwx\" (UID: \"7715e8b4-ae01-4678-a074-0409f721d0d0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:43.053352 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.053285 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc0dfac-2281-4847-a3fe-2d11976315cc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-4qdkd\" (UID: \"6dc0dfac-2281-4847-a3fe-2d11976315cc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:43.053352 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.053315 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4hx\" (UniqueName: \"kubernetes.io/projected/6dc0dfac-2281-4847-a3fe-2d11976315cc-kube-api-access-cl4hx\") pod \"payload-processing-55fbbbf9dc-4qdkd\" (UID: \"6dc0dfac-2281-4847-a3fe-2d11976315cc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:43.081241 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.081214 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:43.154852 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.154751 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc0dfac-2281-4847-a3fe-2d11976315cc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-4qdkd\" (UID: \"6dc0dfac-2281-4847-a3fe-2d11976315cc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:43.154852 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.154794 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4hx\" (UniqueName: \"kubernetes.io/projected/6dc0dfac-2281-4847-a3fe-2d11976315cc-kube-api-access-cl4hx\") pod \"payload-processing-55fbbbf9dc-4qdkd\" (UID: \"6dc0dfac-2281-4847-a3fe-2d11976315cc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:43.155440 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.155421 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc0dfac-2281-4847-a3fe-2d11976315cc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-4qdkd\" (UID: \"6dc0dfac-2281-4847-a3fe-2d11976315cc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:43.213194 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.213165 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4hx\" (UniqueName: \"kubernetes.io/projected/6dc0dfac-2281-4847-a3fe-2d11976315cc-kube-api-access-cl4hx\") pod \"payload-processing-55fbbbf9dc-4qdkd\" (UID: \"6dc0dfac-2281-4847-a3fe-2d11976315cc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:43.241028 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.241005 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:43.443740 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.443608 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx"] Nov 01 03:50:43.446343 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:50:43.446305 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7715e8b4_ae01_4678_a074_0409f721d0d0.slice/crio-8f9ec0cf79801cff4dd567cd36950bb2d5af3f3cb920e290ce1bc597227a671e WatchSource:0}: Error finding container 8f9ec0cf79801cff4dd567cd36950bb2d5af3f3cb920e290ce1bc597227a671e: Status 404 returned error can't find the container with id 8f9ec0cf79801cff4dd567cd36950bb2d5af3f3cb920e290ce1bc597227a671e Nov 01 03:50:43.448313 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.448297 6735 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 03:50:43.604496 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:43.604472 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-4qdkd"] Nov 01 03:50:43.607084 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:50:43.607056 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc0dfac_2281_4847_a3fe_2d11976315cc.slice/crio-35b609c7a302a69ef5183e3f9ee4a9cfd4736e447c0f8030ee10c4c80c7d91db WatchSource:0}: Error finding container 35b609c7a302a69ef5183e3f9ee4a9cfd4736e447c0f8030ee10c4c80c7d91db: Status 404 returned error can't find the container with id 35b609c7a302a69ef5183e3f9ee4a9cfd4736e447c0f8030ee10c4c80c7d91db Nov 01 03:50:44.325738 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:44.325702 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" event={"ID":"6dc0dfac-2281-4847-a3fe-2d11976315cc","Type":"ContainerStarted","Data":"35b609c7a302a69ef5183e3f9ee4a9cfd4736e447c0f8030ee10c4c80c7d91db"} Nov 01 03:50:44.326938 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:44.326910 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" event={"ID":"7715e8b4-ae01-4678-a074-0409f721d0d0","Type":"ContainerStarted","Data":"8f9ec0cf79801cff4dd567cd36950bb2d5af3f3cb920e290ce1bc597227a671e"} Nov 01 03:50:47.340694 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:47.340658 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" event={"ID":"7715e8b4-ae01-4678-a074-0409f721d0d0","Type":"ContainerStarted","Data":"18a944234a7ab58f9cc2c2cee818f5694c6ca1ad839bc22455da3f01abbe1924"} Nov 01 03:50:47.341084 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:47.340799 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:47.450196 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:47.450150 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" podStartSLOduration=2.538290029 podStartE2EDuration="5.45013093s" totalImagesPullingTime="2.911840901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:50:47.449022531 +0000 UTC m=+459.038117160" watchObservedRunningTime="2026-11-01 03:50:47.45013093 +0000 UTC m=+459.039225562" Nov 01 03:50:48.346216 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:48.346179 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" event={"ID":"6dc0dfac-2281-4847-a3fe-2d11976315cc","Type":"ContainerStarted","Data":"2068e2b12cb6483a40730b77e351c480b8ab419e12a6217163d82a6c9da61749"} Nov 01 03:50:48.346606 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:48.346413 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:50:48.458630 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:48.458578 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" podStartSLOduration=2.702096998 podStartE2EDuration="6.458562668s" totalImagesPullingTime="3.75646567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:50:48.458029592 +0000 UTC m=+460.047124220" watchObservedRunningTime="2026-11-01 03:50:48.458562668 +0000 UTC m=+460.047657291" Nov 01 03:50:50.311642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.311615 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:50:50.654098 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.653883 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh"] Nov 01 03:50:50.657515 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.657492 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:50.715554 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.715516 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh"] Nov 01 03:50:50.723845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.723816 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmhl\" (UniqueName: \"kubernetes.io/projected/f4ad6801-8f5a-4405-96e1-da3ee74cccdb-kube-api-access-vfmhl\") pod \"ai-gateway-controller-6bdf4bddbc-hwvmh\" (UID: \"f4ad6801-8f5a-4405-96e1-da3ee74cccdb\") " pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:50.724000 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.723854 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ad6801-8f5a-4405-96e1-da3ee74cccdb-metrics-certs\") pod \"ai-gateway-controller-6bdf4bddbc-hwvmh\" (UID: \"f4ad6801-8f5a-4405-96e1-da3ee74cccdb\") " pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:50.824611 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.824556 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmhl\" (UniqueName: \"kubernetes.io/projected/f4ad6801-8f5a-4405-96e1-da3ee74cccdb-kube-api-access-vfmhl\") pod \"ai-gateway-controller-6bdf4bddbc-hwvmh\" (UID: \"f4ad6801-8f5a-4405-96e1-da3ee74cccdb\") " pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:50.824611 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.824615 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ad6801-8f5a-4405-96e1-da3ee74cccdb-metrics-certs\") pod \"ai-gateway-controller-6bdf4bddbc-hwvmh\" (UID: \"f4ad6801-8f5a-4405-96e1-da3ee74cccdb\") " pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:50.827249 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.827228 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4ad6801-8f5a-4405-96e1-da3ee74cccdb-metrics-certs\") pod \"ai-gateway-controller-6bdf4bddbc-hwvmh\" (UID: \"f4ad6801-8f5a-4405-96e1-da3ee74cccdb\") " pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:50.882678 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.882641 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmhl\" (UniqueName: \"kubernetes.io/projected/f4ad6801-8f5a-4405-96e1-da3ee74cccdb-kube-api-access-vfmhl\") pod \"ai-gateway-controller-6bdf4bddbc-hwvmh\" (UID: \"f4ad6801-8f5a-4405-96e1-da3ee74cccdb\") " pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:50.969795 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:50.969761 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:51.334638 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:51.334557 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh"] Nov 01 03:50:51.337807 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:50:51.337778 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ad6801_8f5a_4405_96e1_da3ee74cccdb.slice/crio-03e985e171372f1ae40ef2bce24578b68bf7e329d09b7b23fd318258110c0883 WatchSource:0}: Error finding container 03e985e171372f1ae40ef2bce24578b68bf7e329d09b7b23fd318258110c0883: Status 404 returned error can't find the container with id 03e985e171372f1ae40ef2bce24578b68bf7e329d09b7b23fd318258110c0883 Nov 01 03:50:51.359490 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:51.359457 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" event={"ID":"f4ad6801-8f5a-4405-96e1-da3ee74cccdb","Type":"ContainerStarted","Data":"03e985e171372f1ae40ef2bce24578b68bf7e329d09b7b23fd318258110c0883"} Nov 01 03:50:52.365550 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:52.365508 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" event={"ID":"f4ad6801-8f5a-4405-96e1-da3ee74cccdb","Type":"ContainerStarted","Data":"cbc6ff043240970f9c62b92f0a0ac38dd565288078ea81f86c88a4d67eaba7b5"} Nov 01 03:50:52.365928 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:52.365576 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:50:52.475795 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:52.475735 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" podStartSLOduration=1.6468575429999999 podStartE2EDuration="2.475720633s" totalImagesPullingTime="828.86309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:50:52.474772151 +0000 UTC m=+464.063866779" watchObservedRunningTime="2026-11-01 03:50:52.475720633 +0000 UTC m=+464.064815253" Nov 01 03:50:58.349693 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:58.349663 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sdhwx" Nov 01 03:50:59.353535 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:50:59.353508 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-4qdkd" Nov 01 03:51:03.000113 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:51:03.000073 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:51:03.371759 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.371674 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6bdf4bddbc-hwvmh" Nov 01 03:51:03.528892 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.528855 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-96g5j"] Nov 01 03:51:03.529161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.529137 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" podUID="1dfc828d-042d-45ee-b7f1-9da4029c0d41" containerName="manager" containerID="cri-o://97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e" gracePeriod=30 Nov 01 03:51:03.765472 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.765449 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:51:03.942718 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.942628 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfc828d-042d-45ee-b7f1-9da4029c0d41-kube-api-access-tvtvg\" (UniqueName: \"kubernetes.io/projected/1dfc828d-042d-45ee-b7f1-9da4029c0d41-kube-api-access-tvtvg\") pod \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " Nov 01 03:51:03.942718 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.942670 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfc828d-042d-45ee-b7f1-9da4029c0d41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfc828d-042d-45ee-b7f1-9da4029c0d41-metrics-certs\") pod \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\" (UID: \"1dfc828d-042d-45ee-b7f1-9da4029c0d41\") " Nov 01 03:51:03.945081 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.945048 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfc828d-042d-45ee-b7f1-9da4029c0d41-kube-api-access-tvtvg" pod "1dfc828d-042d-45ee-b7f1-9da4029c0d41" (UID: "1dfc828d-042d-45ee-b7f1-9da4029c0d41"). InnerVolumeSpecName "kube-api-access-tvtvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:51:03.945218 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:03.945084 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfc828d-042d-45ee-b7f1-9da4029c0d41-metrics-certs" pod "1dfc828d-042d-45ee-b7f1-9da4029c0d41" (UID: "1dfc828d-042d-45ee-b7f1-9da4029c0d41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:51:04.043564 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.043527 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtvg\" (UniqueName: \"kubernetes.io/projected/1dfc828d-042d-45ee-b7f1-9da4029c0d41-kube-api-access-tvtvg\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:51:04.043564 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.043559 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfc828d-042d-45ee-b7f1-9da4029c0d41-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:51:04.417553 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.417513 6735 generic.go:356] "Generic (PLEG): container finished" podID="1dfc828d-042d-45ee-b7f1-9da4029c0d41" containerID="97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e" exitCode=0 Nov 01 03:51:04.417727 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.417560 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" Nov 01 03:51:04.417727 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.417596 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" event={"ID":"1dfc828d-042d-45ee-b7f1-9da4029c0d41","Type":"ContainerDied","Data":"97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e"} Nov 01 03:51:04.417727 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.417651 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-96g5j" event={"ID":"1dfc828d-042d-45ee-b7f1-9da4029c0d41","Type":"ContainerDied","Data":"e2d8180a2137454d8cf2bea540e740eeeca3f73c4bbd25ab3d3f9901851b2f9e"} Nov 01 03:51:04.417727 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.417676 6735 scope.go:122] "RemoveContainer" containerID="97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e" Nov 01 03:51:04.426398 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.426378 6735 scope.go:122] "RemoveContainer" containerID="97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e" Nov 01 03:51:04.426663 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:51:04.426642 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e\": container with ID starting with 97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e not found: ID does not exist" containerID="97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e" Nov 01 03:51:04.426729 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.426676 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e"} err="failed to get container status \"97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e\": rpc error: code = NotFound desc = could not find container \"97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e\": container with ID starting with 97ce7285d7c5bbe3be87f489ae28613025d50be1c5f63dd04e22f7d21aea6e9e not found: ID does not exist" Nov 01 03:51:04.571863 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.571837 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-96g5j"] Nov 01 03:51:04.618228 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:04.618195 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-96g5j"] Nov 01 03:51:06.523638 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:06.523601 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfc828d-042d-45ee-b7f1-9da4029c0d41" path="/var/lib/kubelet/pods/1dfc828d-042d-45ee-b7f1-9da4029c0d41/volumes" Nov 01 03:51:16.010185 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.010139 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n"] Nov 01 03:51:16.010679 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.010646 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1dfc828d-042d-45ee-b7f1-9da4029c0d41" containerName="manager" Nov 01 03:51:16.010679 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.010664 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfc828d-042d-45ee-b7f1-9da4029c0d41" containerName="manager" Nov 01 03:51:16.010790 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.010777 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfc828d-042d-45ee-b7f1-9da4029c0d41" containerName="manager" Nov 01 03:51:16.013867 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.013844 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.035075 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.035045 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.035194 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.035120 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.035245 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.035230 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.035296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.035255 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e51262-5076-48d9-878f-68948911c3b4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.035296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.035277 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lk7\" (UniqueName: \"kubernetes.io/projected/98e51262-5076-48d9-878f-68948911c3b4-kube-api-access-v7lk7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.035419 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.035404 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.109352 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.109316 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 03:51:16.109352 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.109318 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 03:51:16.109352 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.109318 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dpn7r\"" Nov 01 03:51:16.109588 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.109412 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 03:51:16.110335 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.110317 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n"] Nov 01 03:51:16.136329 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136302 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136329 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136330 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136531 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136356 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136531 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136423 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136531 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136447 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e51262-5076-48d9-878f-68948911c3b4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136531 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136470 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lk7\" (UniqueName: \"kubernetes.io/projected/98e51262-5076-48d9-878f-68948911c3b4-kube-api-access-v7lk7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136846 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136823 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136846 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136837 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.136995 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.136904 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.138937 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.138918 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98e51262-5076-48d9-878f-68948911c3b4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.139154 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.139138 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98e51262-5076-48d9-878f-68948911c3b4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.194251 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.194228 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lk7\" (UniqueName: \"kubernetes.io/projected/98e51262-5076-48d9-878f-68948911c3b4-kube-api-access-v7lk7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n\" (UID: \"98e51262-5076-48d9-878f-68948911c3b4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.326902 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.326823 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:16.690296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:16.690206 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n"] Nov 01 03:51:16.693142 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:51:16.693104 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e51262_5076_48d9_878f_68948911c3b4.slice/crio-7d7c590d39175af90aeeead2567ef160e729dd8818ef7ee0309a2f641c5d0a8b WatchSource:0}: Error finding container 7d7c590d39175af90aeeead2567ef160e729dd8818ef7ee0309a2f641c5d0a8b: Status 404 returned error can't find the container with id 7d7c590d39175af90aeeead2567ef160e729dd8818ef7ee0309a2f641c5d0a8b Nov 01 03:51:17.471601 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:17.471566 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" event={"ID":"98e51262-5076-48d9-878f-68948911c3b4","Type":"ContainerStarted","Data":"7d7c590d39175af90aeeead2567ef160e729dd8818ef7ee0309a2f641c5d0a8b"} Nov 01 03:51:19.481946 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:19.481912 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" event={"ID":"98e51262-5076-48d9-878f-68948911c3b4","Type":"ContainerStarted","Data":"739a71abe13b381f34efcac474312bf33ed22bc0d3eb3772f4f32a932e4b5a5b"} Nov 01 03:51:19.592797 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:19.592744 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" podStartSLOduration=2.247230145 podStartE2EDuration="4.592726887s" totalImagesPullingTime="2.345496742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:19.592119267 +0000 UTC m=+491.181213889" watchObservedRunningTime="2026-11-01 03:51:19.592726887 +0000 UTC m=+491.181821552" Nov 01 03:51:19.799592 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:19.799501 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:51:20.327380 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:20.327344 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:21.371276 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.371238 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx"] Nov 01 03:51:21.374936 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.374917 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.424087 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.424059 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 03:51:21.471399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.471368 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx"] Nov 01 03:51:21.484934 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.484897 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.485112 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.484938 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.485112 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.485079 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.485215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.485109 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6zj\" (UniqueName: \"kubernetes.io/projected/a683cab9-8bf5-48e5-9f20-e4d26754b48e-kube-api-access-8c6zj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.485215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.485146 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a683cab9-8bf5-48e5-9f20-e4d26754b48e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.485284 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.485245 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586218 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586183 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a683cab9-8bf5-48e5-9f20-e4d26754b48e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586373 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586228 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586373 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586266 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586373 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586285 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586532 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586371 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586643 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6zj\" (UniqueName: \"kubernetes.io/projected/a683cab9-8bf5-48e5-9f20-e4d26754b48e-kube-api-access-8c6zj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586731 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586764 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.586814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.586808 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.588732 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.588706 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a683cab9-8bf5-48e5-9f20-e4d26754b48e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.588921 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.588902 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a683cab9-8bf5-48e5-9f20-e4d26754b48e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.644442 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.644370 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6zj\" (UniqueName: \"kubernetes.io/projected/a683cab9-8bf5-48e5-9f20-e4d26754b48e-kube-api-access-8c6zj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-22fpx\" (UID: \"a683cab9-8bf5-48e5-9f20-e4d26754b48e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:21.687713 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:21.687685 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:22.053191 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:22.053165 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx"] Nov 01 03:51:22.055808 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:51:22.055779 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda683cab9_8bf5_48e5_9f20_e4d26754b48e.slice/crio-fa2b374328e6cfe7948cc2e2b85de93b1275e1ab56b3186029f973d9bda2ed21 WatchSource:0}: Error finding container fa2b374328e6cfe7948cc2e2b85de93b1275e1ab56b3186029f973d9bda2ed21: Status 404 returned error can't find the container with id fa2b374328e6cfe7948cc2e2b85de93b1275e1ab56b3186029f973d9bda2ed21 Nov 01 03:51:22.497429 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:22.497397 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" event={"ID":"a683cab9-8bf5-48e5-9f20-e4d26754b48e","Type":"ContainerStarted","Data":"fa2b374328e6cfe7948cc2e2b85de93b1275e1ab56b3186029f973d9bda2ed21"} Nov 01 03:51:23.503173 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:23.503140 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" event={"ID":"a683cab9-8bf5-48e5-9f20-e4d26754b48e","Type":"ContainerStarted","Data":"447ec95b33711ef2e0146f895d364f4642264a0bbfa9f1310015a9b7fb9fc0e3"} Nov 01 03:51:23.613855 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:23.613809 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" podStartSLOduration=2.001458759 podStartE2EDuration="2.61379508s" totalImagesPullingTime="612.336321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:23.613678047 +0000 UTC m=+495.202772676" watchObservedRunningTime="2026-11-01 03:51:23.61379508 +0000 UTC m=+495.202889708" Nov 01 03:51:23.687941 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:23.687907 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:26.327700 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:26.327656 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:26.340461 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:26.340429 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:26.527273 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:26.527244 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n" Nov 01 03:51:30.722717 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:30.722682 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:51:31.299519 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.299484 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889"] Nov 01 03:51:31.303133 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.303112 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.352588 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.352558 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 01 03:51:31.360063 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.360021 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889"] Nov 01 03:51:31.374209 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.374183 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a97ea-a619-4e99-b374-bbb6dc85177a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.374351 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.374221 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.374351 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.374255 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.374444 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.374425 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.374488 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.374451 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prggg\" (UniqueName: \"kubernetes.io/projected/ee8a97ea-a619-4e99-b374-bbb6dc85177a-kube-api-access-prggg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.374488 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.374477 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.475642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.475608 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a97ea-a619-4e99-b374-bbb6dc85177a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.475642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.475644 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.475900 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.475666 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.475900 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.475708 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.475900 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.475724 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prggg\" (UniqueName: \"kubernetes.io/projected/ee8a97ea-a619-4e99-b374-bbb6dc85177a-kube-api-access-prggg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.475900 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.475747 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.476168 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.476143 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.476225 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.476172 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.476290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.476244 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.478119 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.478094 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8a97ea-a619-4e99-b374-bbb6dc85177a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.478265 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.478249 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8a97ea-a619-4e99-b374-bbb6dc85177a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.534457 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.534430 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prggg\" (UniqueName: \"kubernetes.io/projected/ee8a97ea-a619-4e99-b374-bbb6dc85177a-kube-api-access-prggg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889\" (UID: \"ee8a97ea-a619-4e99-b374-bbb6dc85177a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.615836 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.615753 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:31.688502 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.688472 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:31.701429 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.701402 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:31.983726 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:31.983697 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889"] Nov 01 03:51:31.986647 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:51:31.986617 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8a97ea_a619_4e99_b374_bbb6dc85177a.slice/crio-e8d3cf81b00c1193a4cdbbd67b2efcacda1a8f68c75b113861e07a5e630a24be WatchSource:0}: Error finding container e8d3cf81b00c1193a4cdbbd67b2efcacda1a8f68c75b113861e07a5e630a24be: Status 404 returned error can't find the container with id e8d3cf81b00c1193a4cdbbd67b2efcacda1a8f68c75b113861e07a5e630a24be Nov 01 03:51:32.540238 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:32.540201 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" event={"ID":"ee8a97ea-a619-4e99-b374-bbb6dc85177a","Type":"ContainerStarted","Data":"e8d3cf81b00c1193a4cdbbd67b2efcacda1a8f68c75b113861e07a5e630a24be"} Nov 01 03:51:32.551804 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:32.551780 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-22fpx" Nov 01 03:51:33.000831 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:51:33.000793 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:51:33.546391 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:33.546356 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" event={"ID":"ee8a97ea-a619-4e99-b374-bbb6dc85177a","Type":"ContainerStarted","Data":"b3d7e2130b9ef24f0ba342e1e01d08051e6b917d93c4772a65b2919106d6dd75"} Nov 01 03:51:33.616583 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:33.616555 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:33.658349 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:33.658292 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" podStartSLOduration=2.057843182 podStartE2EDuration="2.658278422s" totalImagesPullingTime="600.43524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:33.657814123 +0000 UTC m=+505.246908753" watchObservedRunningTime="2026-11-01 03:51:33.658278422 +0000 UTC m=+505.247373051" Nov 01 03:51:36.960839 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:36.960805 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5"] Nov 01 03:51:36.964493 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:36.964473 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.017976 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.017946 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qr8p5\"" Nov 01 03:51:37.024371 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024344 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024454 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024385 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttxp\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-kube-api-access-kttxp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024454 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024408 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dcca89-f3aa-437f-9f17-f525bd108fb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024460 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024529 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024612 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024549 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024612 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024581 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024686 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024623 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.024686 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.024649 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.066293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.066262 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5"] Nov 01 03:51:37.125220 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125185 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttxp\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-kube-api-access-kttxp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125229 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dcca89-f3aa-437f-9f17-f525bd108fb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125278 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125308 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125331 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125355 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125388 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125728 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125422 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125728 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125499 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125825 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125721 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.125904 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125874 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.126005 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125883 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.126005 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.125943 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.126138 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.126121 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dcca89-f3aa-437f-9f17-f525bd108fb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.127698 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.127679 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.128161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.128144 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.186233 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.186199 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.186399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.186330 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttxp\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-kube-api-access-kttxp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n4hh5\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.277198 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.277108 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:37.693645 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:37.693618 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5"] Nov 01 03:51:37.696657 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:51:37.696621 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dcca89_f3aa_437f_9f17_f525bd108fb9.slice/crio-22b74eff0efc9891e776d0c7a0dd1b8dc00ba8a96131f1bb8365691d790cbe5d WatchSource:0}: Error finding container 22b74eff0efc9891e776d0c7a0dd1b8dc00ba8a96131f1bb8365691d790cbe5d: Status 404 returned error can't find the container with id 22b74eff0efc9891e776d0c7a0dd1b8dc00ba8a96131f1bb8365691d790cbe5d Nov 01 03:51:38.126577 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.126539 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh"] Nov 01 03:51:38.131600 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.131570 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.180838 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.180800 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 01 03:51:38.228182 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.228150 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh"] Nov 01 03:51:38.236600 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.236578 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.236706 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.236609 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.236706 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.236626 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.236799 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.236779 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.236833 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.236815 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh2lw\" (UniqueName: \"kubernetes.io/projected/39329c16-0f40-4151-821a-6aac58f565bd-kube-api-access-mh2lw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.236898 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.236884 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39329c16-0f40-4151-821a-6aac58f565bd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.337386 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337355 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh2lw\" (UniqueName: \"kubernetes.io/projected/39329c16-0f40-4151-821a-6aac58f565bd-kube-api-access-mh2lw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.337566 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337401 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39329c16-0f40-4151-821a-6aac58f565bd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.337566 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337440 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.337566 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337459 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.337566 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337473 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.337566 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337518 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.337932 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337905 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.338062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337946 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.338062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.337985 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.339874 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.339856 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39329c16-0f40-4151-821a-6aac58f565bd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.340238 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.340223 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39329c16-0f40-4151-821a-6aac58f565bd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.399482 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.399409 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh2lw\" (UniqueName: \"kubernetes.io/projected/39329c16-0f40-4151-821a-6aac58f565bd-kube-api-access-mh2lw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh\" (UID: \"39329c16-0f40-4151-821a-6aac58f565bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.447022 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.446987 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:38.569320 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.569272 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" event={"ID":"59dcca89-f3aa-437f-9f17-f525bd108fb9","Type":"ContainerStarted","Data":"22b74eff0efc9891e776d0c7a0dd1b8dc00ba8a96131f1bb8365691d790cbe5d"} Nov 01 03:51:38.814902 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:38.814865 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh"] Nov 01 03:51:38.817894 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:51:38.817868 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39329c16_0f40_4151_821a_6aac58f565bd.slice/crio-3db1291d8141d4d388514cbff72eee54a83452afac6719e820aa4168a131ba17 WatchSource:0}: Error finding container 3db1291d8141d4d388514cbff72eee54a83452afac6719e820aa4168a131ba17: Status 404 returned error can't find the container with id 3db1291d8141d4d388514cbff72eee54a83452afac6719e820aa4168a131ba17 Nov 01 03:51:39.575667 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:39.575631 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" event={"ID":"39329c16-0f40-4151-821a-6aac58f565bd","Type":"ContainerStarted","Data":"7043c300d6751ddf711932174922c3516602c34227ac5d45baf7974c8ce7f045"} Nov 01 03:51:39.576074 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:39.575688 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" event={"ID":"39329c16-0f40-4151-821a-6aac58f565bd","Type":"ContainerStarted","Data":"3db1291d8141d4d388514cbff72eee54a83452afac6719e820aa4168a131ba17"} Nov 01 03:51:39.687125 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:39.686998 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" podStartSLOduration=1.10623729 podStartE2EDuration="1.68698282s" totalImagesPullingTime="580.74553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:39.686191503 +0000 UTC m=+511.275286131" watchObservedRunningTime="2026-11-01 03:51:39.68698282 +0000 UTC m=+511.276077444" Nov 01 03:51:40.447440 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:40.447353 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:41.616028 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:41.615993 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:41.629895 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:41.629863 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:42.600656 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:42.600614 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889" Nov 01 03:51:44.354135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:44.354105 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:51:44.354393 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:44.354172 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:51:44.354393 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:44.354203 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:51:44.600803 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:44.600769 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" event={"ID":"59dcca89-f3aa-437f-9f17-f525bd108fb9","Type":"ContainerStarted","Data":"e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf"} Nov 01 03:51:44.718773 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:44.718584 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podStartSLOduration=2.063947165 podStartE2EDuration="8.718473212s" totalImagesPullingTime="6.654526047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:44.716532633 +0000 UTC m=+516.305627262" watchObservedRunningTime="2026-11-01 03:51:44.718473212 +0000 UTC m=+516.307567843" Nov 01 03:51:45.277444 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:45.277401 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:45.279063 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:45.279006 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 03:51:45.279197 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:45.279088 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 03:51:45.279943 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:45.279922 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:51:46.278027 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:46.277989 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 03:51:46.278431 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:46.278078 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 03:51:47.277788 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:47.277744 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 03:51:47.278237 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:47.277806 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 03:51:48.278291 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:48.278255 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 03:51:48.278687 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:48.278327 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 03:51:48.468247 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:48.468211 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:48.481343 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:48.481313 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:48.629393 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:48.629313 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh" Nov 01 03:51:49.277511 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:49.277480 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 03:51:49.277689 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:49.277545 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 03:51:50.278111 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:50.278076 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 03:51:50.278468 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:50.278142 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 03:51:51.278303 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:51.278265 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 01 03:51:51.278730 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:51.278333 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 01 03:51:52.094944 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.094907 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899"] Nov 01 03:51:52.101461 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.101436 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174141 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174107 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174141 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174154 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174362 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174181 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174362 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174240 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b554f3ac-a04c-40bb-b366-ea47151938ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174362 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174328 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174362 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174353 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174492 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174383 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174492 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174473 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4xr\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-kube-api-access-2f4xr\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.174564 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.174517 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.199259 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.199223 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899"] Nov 01 03:51:52.275006 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.274968 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4xr\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-kube-api-access-2f4xr\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275231 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275016 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275231 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275100 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275332 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275237 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275332 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275305 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275433 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275330 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b554f3ac-a04c-40bb-b366-ea47151938ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275433 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275380 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275433 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275411 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275593 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275460 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275738 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275707 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275738 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275725 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.275907 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.275786 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.276246 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.276126 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.276339 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.276247 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b554f3ac-a04c-40bb-b366-ea47151938ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.277533 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.277503 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:52.277712 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.277690 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.277891 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.277872 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.332697 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.332663 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.333012 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.332797 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4xr\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-kube-api-access-2f4xr\") pod \"maas-default-gateway-openshift-default-5df48fcc99-b8899\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.414640 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.414562 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:52.833272 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.833242 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899"] Nov 01 03:51:52.837194 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:51:52.837164 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb554f3ac_a04c_40bb_b366_ea47151938ad.slice/crio-3558ba7a01a6f0140a487d5756098ed19582d80ed0d5804d2026d396c8ee290e WatchSource:0}: Error finding container 3558ba7a01a6f0140a487d5756098ed19582d80ed0d5804d2026d396c8ee290e: Status 404 returned error can't find the container with id 3558ba7a01a6f0140a487d5756098ed19582d80ed0d5804d2026d396c8ee290e Nov 01 03:51:52.840327 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.840292 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:51:52.840414 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.840359 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:51:52.840414 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:52.840388 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:51:53.319562 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:53.319500 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 03:51:53.319869 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:53.319831 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 03:51:53.675116 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:53.674594 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" event={"ID":"b554f3ac-a04c-40bb-b366-ea47151938ad","Type":"ContainerStarted","Data":"204a5dc4cc41966e4620ba59d75a1ceef03c06c355e16b6a33b4a27dea3349cb"} Nov 01 03:51:53.675116 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:53.674672 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" event={"ID":"b554f3ac-a04c-40bb-b366-ea47151938ad","Type":"ContainerStarted","Data":"3558ba7a01a6f0140a487d5756098ed19582d80ed0d5804d2026d396c8ee290e"} Nov 01 03:51:53.840759 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:53.840703 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podStartSLOduration=1.840687713 podStartE2EDuration="1.840687713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:53.839333812 +0000 UTC m=+525.428428452" watchObservedRunningTime="2026-11-01 03:51:53.840687713 +0000 UTC m=+525.429782341" Nov 01 03:51:54.367225 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:54.367173 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n4hh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 03:51:54.367523 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:54.367497 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 03:51:54.415653 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:54.415624 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:51:54.415809 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:54.415746 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:54.416750 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:54.416729 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:51:54.417139 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:54.417113 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-b8899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 03:51:54.417201 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:54.417165 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 03:51:55.415400 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:55.415364 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-b8899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 03:51:55.415803 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:55.415420 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 03:51:56.415817 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:56.415784 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-b8899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 03:51:56.416271 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:56.415839 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 03:51:57.415126 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:57.415088 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-b8899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 03:51:57.417212 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:57.415148 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 03:51:58.414967 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:58.414937 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-b8899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 03:51:58.415185 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:58.414986 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 03:51:59.415351 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:59.415314 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-b8899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 03:51:59.415822 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:51:59.415376 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 03:52:01.059984 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:01.059953 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:52:01.060516 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:01.060403 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:52:01.061305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:01.061283 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 03:52:03.000172 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:52:03.000133 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:52:33.000635 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:52:33.000554 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:52:42.069600 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.069565 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf"] Nov 01 03:52:42.073467 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.073445 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.123346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.123316 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4c476fff-openshift-default-dockercfg-2b24m\"" Nov 01 03:52:42.170996 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.170968 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf"] Nov 01 03:52:42.238342 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238309 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-socket\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238342 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238342 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-credential-socket\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238368 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-certs\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238487 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-podinfo\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238526 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-data\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238665 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238559 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-token\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238704 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238683 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istiod-ca-cert\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238737 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238708 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-envoy\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.238737 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.238725 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5pmj\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-kube-api-access-w5pmj\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339506 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339425 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-token\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339506 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339505 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istiod-ca-cert\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339747 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339527 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-envoy\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339747 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339543 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pmj\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-kube-api-access-w5pmj\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339747 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339571 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-socket\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339747 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339590 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-credential-socket\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339747 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339621 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-certs\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339747 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339657 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-podinfo\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.339747 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.339692 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-data\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.340155 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.340134 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-data\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.340223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.340155 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-socket\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.340223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.340180 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-credential-socket\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.340331 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.340281 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-certs\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.340580 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.340557 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istiod-ca-cert\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.342009 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.341971 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-envoy\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.342296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.342274 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-podinfo\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.425198 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.425163 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pmj\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-kube-api-access-w5pmj\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.426888 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.426844 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-token\") pod \"e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:42.686090 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:42.685982 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:43.108449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:43.108421 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf"] Nov 01 03:52:43.110764 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:52:43.110732 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7ed28a_43ad_4a96_ba1b_5a5b0b2ae42f.slice/crio-10119f1557ea6d71e25080b7267e50108f39dd7e01692282b31a97c54451698e WatchSource:0}: Error finding container 10119f1557ea6d71e25080b7267e50108f39dd7e01692282b31a97c54451698e: Status 404 returned error can't find the container with id 10119f1557ea6d71e25080b7267e50108f39dd7e01692282b31a97c54451698e Nov 01 03:52:43.112956 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:43.112921 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:52:43.113051 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:43.112992 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:52:43.113051 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:43.113024 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:52:43.905962 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:43.905927 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" event={"ID":"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f","Type":"ContainerStarted","Data":"504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f"} Nov 01 03:52:43.906148 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:43.905978 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" event={"ID":"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f","Type":"ContainerStarted","Data":"10119f1557ea6d71e25080b7267e50108f39dd7e01692282b31a97c54451698e"} Nov 01 03:52:44.074169 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.074115 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podStartSLOduration=2.074101631 podStartE2EDuration="2.074101631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:44.066512334 +0000 UTC m=+575.655606962" watchObservedRunningTime="2026-11-01 03:52:44.074101631 +0000 UTC m=+575.663196260" Nov 01 03:52:44.114065 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.114006 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4"] Nov 01 03:52:44.117774 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.117754 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.169365 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.169283 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3145c214-openshift-default-dockercfg-sk8f5\"" Nov 01 03:52:44.185170 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.185132 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4"] Nov 01 03:52:44.260334 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260215 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-certs\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.260580 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260355 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-socket\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.260580 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260433 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-podinfo\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.260580 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260468 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-token\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.260580 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260495 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-data\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.260580 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260575 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-credential-socket\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.261270 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260608 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzzv\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-kube-api-access-2gzzv\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.261270 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260640 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istiod-ca-cert\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.261270 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.260680 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-envoy\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.367806 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.367759 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-podinfo\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.367971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.367825 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-token\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.367971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.367853 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-data\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.367971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.367934 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-credential-socket\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.367971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.367962 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzzv\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-kube-api-access-2gzzv\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.368194 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.367993 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istiod-ca-cert\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.368194 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.368060 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-envoy\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.368194 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.368096 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-certs\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.368194 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.368130 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-socket\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.369302 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.368632 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-socket\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.369500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.369475 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-data\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.369629 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.369506 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-credential-socket\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.370062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.370011 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-certs\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.370228 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.370206 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istiod-ca-cert\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.372325 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.372300 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-envoy\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.372484 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.372465 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-podinfo\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.431748 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.431669 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-token\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.431873 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.431778 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzzv\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-kube-api-access-2gzzv\") pod \"e2e-ait-3145c214-openshift-default-75d84f979f-tfls4\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.432454 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.432428 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:44.686391 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.686309 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:44.691890 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.691863 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:44.849840 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.849805 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4"] Nov 01 03:52:44.856708 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:52:44.856679 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17aabe7d_4f58_4f9a_ae2f_7f0e79e57fe2.slice/crio-604aa6362923259cae25bff6a22f00f2fd69b0ac1bb408bdcf60caabcd2cea3f WatchSource:0}: Error finding container 604aa6362923259cae25bff6a22f00f2fd69b0ac1bb408bdcf60caabcd2cea3f: Status 404 returned error can't find the container with id 604aa6362923259cae25bff6a22f00f2fd69b0ac1bb408bdcf60caabcd2cea3f Nov 01 03:52:44.859014 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.858977 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:52:44.859093 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.859060 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:52:44.859129 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.859095 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:52:44.912743 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.912718 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" event={"ID":"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2","Type":"ContainerStarted","Data":"604aa6362923259cae25bff6a22f00f2fd69b0ac1bb408bdcf60caabcd2cea3f"} Nov 01 03:52:44.912957 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.912936 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:44.914087 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:44.914064 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:52:45.919187 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:45.919151 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" event={"ID":"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2","Type":"ContainerStarted","Data":"f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d"} Nov 01 03:52:46.125950 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:46.125896 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podStartSLOduration=2.125881493 podStartE2EDuration="2.125881493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:46.079329003 +0000 UTC m=+577.668423632" watchObservedRunningTime="2026-11-01 03:52:46.125881493 +0000 UTC m=+577.714976179" Nov 01 03:52:46.433373 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:46.433334 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:46.438285 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:46.438260 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:46.924636 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:46.924606 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:46.925613 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:46.925591 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:52:49.315659 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.315618 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh"] Nov 01 03:52:49.319336 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.319317 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.420063 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.418154 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-48d2fd87\"" Nov 01 03:52:49.420063 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.418282 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-48d2fd87-dockercfg-zqzn4\"" Nov 01 03:52:49.426296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.420997 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh"] Nov 01 03:52:49.426296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.421680 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvr5\" (UniqueName: \"kubernetes.io/projected/592b33e0-0019-4265-b354-fa4c57e496f6-kube-api-access-vtvr5\") pod \"payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.426296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.421801 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592b33e0-0019-4265-b354-fa4c57e496f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.516762 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.516731 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt"] Nov 01 03:52:49.520979 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.520960 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:49.522418 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.522394 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvr5\" (UniqueName: \"kubernetes.io/projected/592b33e0-0019-4265-b354-fa4c57e496f6-kube-api-access-vtvr5\") pod \"payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.522541 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.522444 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592b33e0-0019-4265-b354-fa4c57e496f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.523373 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.523351 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592b33e0-0019-4265-b354-fa4c57e496f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.578472 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.578398 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt"] Nov 01 03:52:49.583934 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.583898 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvr5\" (UniqueName: \"kubernetes.io/projected/592b33e0-0019-4265-b354-fa4c57e496f6-kube-api-access-vtvr5\") pod \"payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.624107 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.624074 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f8e02-aa6e-4c7e-8686-fd82abe22e58-plugins-config-volume\") pod \"payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:49.624358 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.624326 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j48t7\" (UniqueName: \"kubernetes.io/projected/216f8e02-aa6e-4c7e-8686-fd82abe22e58-kube-api-access-j48t7\") pod \"payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:49.631748 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.631726 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:49.725727 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.725688 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j48t7\" (UniqueName: \"kubernetes.io/projected/216f8e02-aa6e-4c7e-8686-fd82abe22e58-kube-api-access-j48t7\") pod \"payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:49.725877 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.725775 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f8e02-aa6e-4c7e-8686-fd82abe22e58-plugins-config-volume\") pod \"payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:49.726479 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.726459 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f8e02-aa6e-4c7e-8686-fd82abe22e58-plugins-config-volume\") pod \"payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:49.784340 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.784308 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j48t7\" (UniqueName: \"kubernetes.io/projected/216f8e02-aa6e-4c7e-8686-fd82abe22e58-kube-api-access-j48t7\") pod \"payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:49.835402 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:49.835325 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:50.000849 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.000821 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh"] Nov 01 03:52:50.002663 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:52:50.002632 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592b33e0_0019_4265_b354_fa4c57e496f6.slice/crio-2246c9fd8fe0320d489774a1ee2c05fb9a5a7b0de786b577069dbd4c9c945929 WatchSource:0}: Error finding container 2246c9fd8fe0320d489774a1ee2c05fb9a5a7b0de786b577069dbd4c9c945929: Status 404 returned error can't find the container with id 2246c9fd8fe0320d489774a1ee2c05fb9a5a7b0de786b577069dbd4c9c945929 Nov 01 03:52:50.202346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.202319 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt"] Nov 01 03:52:50.204893 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:52:50.204859 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f8e02_aa6e_4c7e_8686_fd82abe22e58.slice/crio-2dbbde4e0708e02b0c4cf604ea638542b095b35e35620c19af5a066fe20ceba8 WatchSource:0}: Error finding container 2dbbde4e0708e02b0c4cf604ea638542b095b35e35620c19af5a066fe20ceba8: Status 404 returned error can't find the container with id 2dbbde4e0708e02b0c4cf604ea638542b095b35e35620c19af5a066fe20ceba8 Nov 01 03:52:50.944745 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.944701 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" event={"ID":"592b33e0-0019-4265-b354-fa4c57e496f6","Type":"ContainerStarted","Data":"cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023"} Nov 01 03:52:50.945242 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.944759 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" event={"ID":"592b33e0-0019-4265-b354-fa4c57e496f6","Type":"ContainerStarted","Data":"2246c9fd8fe0320d489774a1ee2c05fb9a5a7b0de786b577069dbd4c9c945929"} Nov 01 03:52:50.945242 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.944785 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:52:50.946316 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.946284 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" event={"ID":"216f8e02-aa6e-4c7e-8686-fd82abe22e58","Type":"ContainerStarted","Data":"058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7"} Nov 01 03:52:50.946429 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.946324 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" event={"ID":"216f8e02-aa6e-4c7e-8686-fd82abe22e58","Type":"ContainerStarted","Data":"2dbbde4e0708e02b0c4cf604ea638542b095b35e35620c19af5a066fe20ceba8"} Nov 01 03:52:50.946429 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:50.946359 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:52:51.057831 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.057776 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" podStartSLOduration=2.057762166 podStartE2EDuration="2.057762166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:51.057112276 +0000 UTC m=+582.646206906" watchObservedRunningTime="2026-11-01 03:52:51.057762166 +0000 UTC m=+582.646856791" Nov 01 03:52:51.801532 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.801482 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" podStartSLOduration=2.801465867 podStartE2EDuration="2.801465867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:51.166087516 +0000 UTC m=+582.755182160" watchObservedRunningTime="2026-11-01 03:52:51.801465867 +0000 UTC m=+583.390560496" Nov 01 03:52:51.801911 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.801892 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg"] Nov 01 03:52:51.805594 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.805573 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.846311 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.846284 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-metrics-certs\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.846449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.846324 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d29490-451d-4636-ac7d-bfd4e1a0f301-openshift-service-ca\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.846449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.846426 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.846539 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.846499 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9687\" (UniqueName: \"kubernetes.io/projected/12d29490-451d-4636-ac7d-bfd4e1a0f301-kube-api-access-d9687\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.854679 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.854651 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4c476fff-serving-cert\"" Nov 01 03:52:51.902777 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.902747 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg"] Nov 01 03:52:51.947722 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.947684 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-metrics-certs\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.947722 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.947734 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d29490-451d-4636-ac7d-bfd4e1a0f301-openshift-service-ca\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.948305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.947791 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.948305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.947881 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9687\" (UniqueName: \"kubernetes.io/projected/12d29490-451d-4636-ac7d-bfd4e1a0f301-kube-api-access-d9687\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.948564 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:52:51.948533 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-serving-cert: secret "maas-api-e2e-ait-4c476fff-serving-cert" not found Nov 01 03:52:51.948749 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:52:51.948719 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls podName:12d29490-451d-4636-ac7d-bfd4e1a0f301 nodeName:}" failed. No retries permitted until 2026-11-01 03:52:52.44869621 +0000 UTC m=+584.037790835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls") pod "maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" (UID: "12d29490-451d-4636-ac7d-bfd4e1a0f301") : secret "maas-api-e2e-ait-4c476fff-serving-cert" not found Nov 01 03:52:51.949079 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.949055 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d29490-451d-4636-ac7d-bfd4e1a0f301-openshift-service-ca\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:51.954647 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:51.954623 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-metrics-certs\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:52.006895 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:52.006868 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9687\" (UniqueName: \"kubernetes.io/projected/12d29490-451d-4636-ac7d-bfd4e1a0f301-kube-api-access-d9687\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:52.452368 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:52.452331 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:52.454870 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:52.454851 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls\") pod \"maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:52.719214 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:52.719186 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:53.085053 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:53.085007 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg"] Nov 01 03:52:53.087850 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:52:53.087825 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d29490_451d_4636_ac7d_bfd4e1a0f301.slice/crio-0f7bf3d77ddbc2845ab4214f5f55a1608672405284401689f9c8e30844c64a92 WatchSource:0}: Error finding container 0f7bf3d77ddbc2845ab4214f5f55a1608672405284401689f9c8e30844c64a92: Status 404 returned error can't find the container with id 0f7bf3d77ddbc2845ab4214f5f55a1608672405284401689f9c8e30844c64a92 Nov 01 03:52:53.964350 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:53.964311 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" event={"ID":"12d29490-451d-4636-ac7d-bfd4e1a0f301","Type":"ContainerStarted","Data":"0f7bf3d77ddbc2845ab4214f5f55a1608672405284401689f9c8e30844c64a92"} Nov 01 03:52:54.970188 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:54.970144 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" event={"ID":"12d29490-451d-4636-ac7d-bfd4e1a0f301","Type":"ContainerStarted","Data":"cc5389bbb70268814a91c2357a3ddcf7ae23459c51b99cb846e6c5bcb0ea1b5e"} Nov 01 03:52:54.970575 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:54.970266 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:52:55.081173 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:55.081118 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" podStartSLOduration=3.03299752 podStartE2EDuration="4.081105502s" totalImagesPullingTime="1.048107982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:55.080433769 +0000 UTC m=+586.669528400" watchObservedRunningTime="2026-11-01 03:52:55.081105502 +0000 UTC m=+586.670200130" Nov 01 03:52:56.453101 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.452923 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6"] Nov 01 03:52:56.457941 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.457821 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:56.488456 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.488427 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3018d527-9060-4621-a640-39f5a8fc9ec8-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:56.488597 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.488463 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr57n\" (UniqueName: \"kubernetes.io/projected/3018d527-9060-4621-a640-39f5a8fc9ec8-kube-api-access-xr57n\") pod \"payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:56.512812 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.512786 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6"] Nov 01 03:52:56.554773 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.554750 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4c476fff\"" Nov 01 03:52:56.554903 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.554797 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4c476fff-dockercfg-xq4l7\"" Nov 01 03:52:56.589020 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.588988 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3018d527-9060-4621-a640-39f5a8fc9ec8-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:56.589020 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.589021 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr57n\" (UniqueName: \"kubernetes.io/projected/3018d527-9060-4621-a640-39f5a8fc9ec8-kube-api-access-xr57n\") pod \"payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:56.589772 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.589747 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3018d527-9060-4621-a640-39f5a8fc9ec8-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:56.648744 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.648717 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr57n\" (UniqueName: \"kubernetes.io/projected/3018d527-9060-4621-a640-39f5a8fc9ec8-kube-api-access-xr57n\") pod \"payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:56.777077 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:56.777014 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:57.142750 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:57.142726 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6"] Nov 01 03:52:57.145382 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:52:57.145356 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3018d527_9060_4621_a640_39f5a8fc9ec8.slice/crio-cf0aa96a89d2e0f4f507d2aca1a2e383409e3de4cd5748fbd1ebee13dc81ddb4 WatchSource:0}: Error finding container cf0aa96a89d2e0f4f507d2aca1a2e383409e3de4cd5748fbd1ebee13dc81ddb4: Status 404 returned error can't find the container with id cf0aa96a89d2e0f4f507d2aca1a2e383409e3de4cd5748fbd1ebee13dc81ddb4 Nov 01 03:52:58.009959 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:58.009919 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" event={"ID":"3018d527-9060-4621-a640-39f5a8fc9ec8","Type":"ContainerStarted","Data":"e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a"} Nov 01 03:52:58.010434 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:58.009996 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" event={"ID":"3018d527-9060-4621-a640-39f5a8fc9ec8","Type":"ContainerStarted","Data":"cf0aa96a89d2e0f4f507d2aca1a2e383409e3de4cd5748fbd1ebee13dc81ddb4"} Nov 01 03:52:58.010434 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:58.010149 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:52:58.160420 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:52:58.160368 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" podStartSLOduration=2.1603531 podStartE2EDuration="2.1603531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:58.120939442 +0000 UTC m=+589.710034082" watchObservedRunningTime="2026-11-01 03:52:58.1603531 +0000 UTC m=+589.749447741" Nov 01 03:53:01.001872 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:01.001845 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:53:01.348428 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:01.348351 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:53:01.955290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:01.955264 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:53:01.955465 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:01.955341 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:53:02.684588 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.682365 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2"] Nov 01 03:53:02.687112 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.687076 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.785880 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.785846 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f50c3be9-serving-cert\"" Nov 01 03:53:02.787419 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.787394 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2"] Nov 01 03:53:02.865073 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.863808 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48318085-4510-4ad4-81fa-8f0fa77f4029-openshift-service-ca\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.865073 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.863877 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-metrics-certs\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.865073 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.863942 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-maas-api-tls\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.865073 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.864009 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bz2k\" (UniqueName: \"kubernetes.io/projected/48318085-4510-4ad4-81fa-8f0fa77f4029-kube-api-access-4bz2k\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.970061 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.966803 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48318085-4510-4ad4-81fa-8f0fa77f4029-openshift-service-ca\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.970061 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.966876 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-metrics-certs\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.970061 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.966937 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-maas-api-tls\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.970061 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.967000 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bz2k\" (UniqueName: \"kubernetes.io/projected/48318085-4510-4ad4-81fa-8f0fa77f4029-kube-api-access-4bz2k\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.970061 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.968149 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48318085-4510-4ad4-81fa-8f0fa77f4029-openshift-service-ca\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.971953 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.971925 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-metrics-certs\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:02.972584 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:02.972555 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-maas-api-tls\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:03.002345 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:53:03.002307 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:53:03.025503 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:03.025479 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bz2k\" (UniqueName: \"kubernetes.io/projected/48318085-4510-4ad4-81fa-8f0fa77f4029-kube-api-access-4bz2k\") pod \"maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:03.301310 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:03.301222 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:03.688158 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:03.688018 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2"] Nov 01 03:53:03.691063 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:53:03.691015 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48318085_4510_4ad4_81fa_8f0fa77f4029.slice/crio-910dea5660a78cb91c7bad06b088c4a831ff43c378df36c248832c17489d63cb WatchSource:0}: Error finding container 910dea5660a78cb91c7bad06b088c4a831ff43c378df36c248832c17489d63cb: Status 404 returned error can't find the container with id 910dea5660a78cb91c7bad06b088c4a831ff43c378df36c248832c17489d63cb Nov 01 03:53:04.069422 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:04.067463 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" event={"ID":"48318085-4510-4ad4-81fa-8f0fa77f4029","Type":"ContainerStarted","Data":"910dea5660a78cb91c7bad06b088c4a831ff43c378df36c248832c17489d63cb"} Nov 01 03:53:05.077551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:05.077516 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" event={"ID":"48318085-4510-4ad4-81fa-8f0fa77f4029","Type":"ContainerStarted","Data":"94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2"} Nov 01 03:53:05.077930 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:05.077595 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:05.189776 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:05.189718 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" podStartSLOduration=2.302035074 podStartE2EDuration="3.189700804s" totalImagesPullingTime="887.66573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:05.188298892 +0000 UTC m=+596.777393524" watchObservedRunningTime="2026-11-01 03:53:05.189700804 +0000 UTC m=+596.778795433" Nov 01 03:53:08.572556 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:08.572523 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:53:08.573862 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:08.573840 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:53:09.025428 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:09.025396 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:53:09.950856 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:09.950821 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:53:11.088927 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:11.088848 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:11.107873 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:11.107843 6735 generic.go:356] "Generic (PLEG): container finished" podID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerID="504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f" exitCode=0 Nov 01 03:53:11.108021 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:11.107916 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" event={"ID":"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f","Type":"ContainerDied","Data":"504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f"} Nov 01 03:53:11.108293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:11.108278 6735 scope.go:122] "RemoveContainer" containerID="504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f" Nov 01 03:53:11.109621 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:11.109585 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:53:11.109705 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:11.109686 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:53:11.109760 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:11.109744 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:53:12.115375 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:12.115335 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" event={"ID":"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f","Type":"ContainerStarted","Data":"c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08"} Nov 01 03:53:12.688436 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:12.688406 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:53:12.688795 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:12.688772 6735 patch_prober.go:29] interesting pod/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 03:53:12.688946 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:12.688922 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 03:53:13.689062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:13.686451 6735 patch_prober.go:29] interesting pod/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 03:53:13.689062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:13.686515 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 03:53:14.686607 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:14.686550 6735 patch_prober.go:29] interesting pod/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 03:53:14.686929 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:14.686879 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 03:53:14.913497 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:14.913462 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:53:15.695458 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:15.695427 6735 patch_prober.go:29] interesting pod/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 03:53:15.695727 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:15.695704 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 03:53:16.686567 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:16.686524 6735 patch_prober.go:29] interesting pod/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 03:53:16.687057 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:16.686587 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 03:53:17.691498 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:17.691453 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:53:17.693051 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:17.693008 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:53:17.966203 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:17.966168 6735 patch_prober.go:29] interesting pod/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 03:53:17.966392 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:17.966229 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 03:53:20.167947 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:20.167908 6735 generic.go:356] "Generic (PLEG): container finished" podID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerID="f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d" exitCode=0 Nov 01 03:53:20.168482 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:20.167969 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" event={"ID":"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2","Type":"ContainerDied","Data":"f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d"} Nov 01 03:53:20.168482 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:20.168389 6735 scope.go:122] "RemoveContainer" containerID="f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d" Nov 01 03:53:20.169745 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:20.169713 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:53:20.169854 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:20.169800 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:53:20.169921 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:20.169874 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:53:21.177051 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:21.177014 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" event={"ID":"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2","Type":"ContainerStarted","Data":"0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859"} Nov 01 03:53:21.433333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:21.433249 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:53:21.433500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:21.433482 6735 patch_prober.go:29] interesting pod/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 03:53:21.433546 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:21.433523 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 03:53:22.433236 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:22.433200 6735 patch_prober.go:29] interesting pod/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 03:53:22.433710 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:22.433262 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 03:53:23.433171 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:23.433124 6735 patch_prober.go:29] interesting pod/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 03:53:23.433366 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:23.433191 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 03:53:24.433399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:24.433367 6735 patch_prober.go:29] interesting pod/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 03:53:24.433778 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:24.433427 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 03:53:25.280461 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:25.280421 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:53:25.432957 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:25.432921 6735 patch_prober.go:29] interesting pod/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 01 03:53:25.433168 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:25.432989 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 01 03:53:26.437002 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:26.436975 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:53:26.437423 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:26.437377 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:53:26.438145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:26.438124 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:53:33.000389 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:53:33.000354 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:53:45.766569 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:45.766533 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6"] Nov 01 03:53:50.766874 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:50.766808 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" podUID="3018d527-9060-4621-a640-39f5a8fc9ec8" containerName="payload-processing" containerID="cri-o://e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a" gracePeriod=25 Nov 01 03:53:51.005339 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.005308 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:53:51.145110 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.145007 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3018d527-9060-4621-a640-39f5a8fc9ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3018d527-9060-4621-a640-39f5a8fc9ec8-plugins-config-volume\") pod \"3018d527-9060-4621-a640-39f5a8fc9ec8\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " Nov 01 03:53:51.145293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.145192 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3018d527-9060-4621-a640-39f5a8fc9ec8-kube-api-access-xr57n\" (UniqueName: \"kubernetes.io/projected/3018d527-9060-4621-a640-39f5a8fc9ec8-kube-api-access-xr57n\") pod \"3018d527-9060-4621-a640-39f5a8fc9ec8\" (UID: \"3018d527-9060-4621-a640-39f5a8fc9ec8\") " Nov 01 03:53:51.145397 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.145369 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3018d527-9060-4621-a640-39f5a8fc9ec8-plugins-config-volume" pod "3018d527-9060-4621-a640-39f5a8fc9ec8" (UID: "3018d527-9060-4621-a640-39f5a8fc9ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:53:51.147359 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.147332 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3018d527-9060-4621-a640-39f5a8fc9ec8-kube-api-access-xr57n" pod "3018d527-9060-4621-a640-39f5a8fc9ec8" (UID: "3018d527-9060-4621-a640-39f5a8fc9ec8"). InnerVolumeSpecName "kube-api-access-xr57n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:53:51.246748 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.246716 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr57n\" (UniqueName: \"kubernetes.io/projected/3018d527-9060-4621-a640-39f5a8fc9ec8-kube-api-access-xr57n\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:51.246748 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.246743 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3018d527-9060-4621-a640-39f5a8fc9ec8-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:51.334025 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.333992 6735 generic.go:356] "Generic (PLEG): container finished" podID="3018d527-9060-4621-a640-39f5a8fc9ec8" containerID="e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a" exitCode=0 Nov 01 03:53:51.334208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.334067 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" event={"ID":"3018d527-9060-4621-a640-39f5a8fc9ec8","Type":"ContainerDied","Data":"e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a"} Nov 01 03:53:51.334208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.334148 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" event={"ID":"3018d527-9060-4621-a640-39f5a8fc9ec8","Type":"ContainerDied","Data":"cf0aa96a89d2e0f4f507d2aca1a2e383409e3de4cd5748fbd1ebee13dc81ddb4"} Nov 01 03:53:51.334208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.334077 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6" Nov 01 03:53:51.334208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.334174 6735 scope.go:122] "RemoveContainer" containerID="e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a" Nov 01 03:53:51.344075 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.344028 6735 scope.go:122] "RemoveContainer" containerID="e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a" Nov 01 03:53:51.344337 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:53:51.344318 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a\": container with ID starting with e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a not found: ID does not exist" containerID="e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a" Nov 01 03:53:51.344404 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.344344 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a"} err="failed to get container status \"e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a\": rpc error: code = NotFound desc = could not find container \"e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a\": container with ID starting with e92a4ac85bdb87e78c88fd7741f62817ae0ce433b6cdddc896539397629c5a8a not found: ID does not exist" Nov 01 03:53:51.491577 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.491543 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6"] Nov 01 03:53:51.497852 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:51.497826 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c476fff-74cfd5968c-htfq6"] Nov 01 03:53:52.523856 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:52.523823 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3018d527-9060-4621-a640-39f5a8fc9ec8" path="/var/lib/kubelet/pods/3018d527-9060-4621-a640-39f5a8fc9ec8/volumes" Nov 01 03:53:54.206209 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.206170 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg"] Nov 01 03:53:54.206675 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.206436 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" podUID="12d29490-451d-4636-ac7d-bfd4e1a0f301" containerName="maas-api" containerID="cri-o://cc5389bbb70268814a91c2357a3ddcf7ae23459c51b99cb846e6c5bcb0ea1b5e" gracePeriod=30 Nov 01 03:53:54.352285 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.352250 6735 generic.go:356] "Generic (PLEG): container finished" podID="12d29490-451d-4636-ac7d-bfd4e1a0f301" containerID="cc5389bbb70268814a91c2357a3ddcf7ae23459c51b99cb846e6c5bcb0ea1b5e" exitCode=0 Nov 01 03:53:54.352447 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.352288 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" event={"ID":"12d29490-451d-4636-ac7d-bfd4e1a0f301","Type":"ContainerDied","Data":"cc5389bbb70268814a91c2357a3ddcf7ae23459c51b99cb846e6c5bcb0ea1b5e"} Nov 01 03:53:54.478630 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.478598 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:53:54.607296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.595550 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d29490-451d-4636-ac7d-bfd4e1a0f301-kube-api-access-d9687\" (UniqueName: \"kubernetes.io/projected/12d29490-451d-4636-ac7d-bfd4e1a0f301-kube-api-access-d9687\") pod \"12d29490-451d-4636-ac7d-bfd4e1a0f301\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " Nov 01 03:53:54.607296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.595621 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls\") pod \"12d29490-451d-4636-ac7d-bfd4e1a0f301\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " Nov 01 03:53:54.607296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.595690 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-metrics-certs\") pod \"12d29490-451d-4636-ac7d-bfd4e1a0f301\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " Nov 01 03:53:54.607296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.595735 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d29490-451d-4636-ac7d-bfd4e1a0f301-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d29490-451d-4636-ac7d-bfd4e1a0f301-openshift-service-ca\") pod \"12d29490-451d-4636-ac7d-bfd4e1a0f301\" (UID: \"12d29490-451d-4636-ac7d-bfd4e1a0f301\") " Nov 01 03:53:54.607296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.596437 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d29490-451d-4636-ac7d-bfd4e1a0f301-openshift-service-ca" pod "12d29490-451d-4636-ac7d-bfd4e1a0f301" (UID: "12d29490-451d-4636-ac7d-bfd4e1a0f301"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:53:54.607296 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.605914 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d29490-451d-4636-ac7d-bfd4e1a0f301-kube-api-access-d9687" pod "12d29490-451d-4636-ac7d-bfd4e1a0f301" (UID: "12d29490-451d-4636-ac7d-bfd4e1a0f301"). InnerVolumeSpecName "kube-api-access-d9687". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:53:54.611058 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.608536 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls" pod "12d29490-451d-4636-ac7d-bfd4e1a0f301" (UID: "12d29490-451d-4636-ac7d-bfd4e1a0f301"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:53:54.611868 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.611822 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-metrics-certs" pod "12d29490-451d-4636-ac7d-bfd4e1a0f301" (UID: "12d29490-451d-4636-ac7d-bfd4e1a0f301"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:53:54.697480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.697439 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9687\" (UniqueName: \"kubernetes.io/projected/12d29490-451d-4636-ac7d-bfd4e1a0f301-kube-api-access-d9687\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:54.697480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.697478 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:54.697683 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.697494 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d29490-451d-4636-ac7d-bfd4e1a0f301-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:54.697683 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:54.697510 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d29490-451d-4636-ac7d-bfd4e1a0f301-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:55.358086 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:55.358014 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" event={"ID":"12d29490-451d-4636-ac7d-bfd4e1a0f301","Type":"ContainerDied","Data":"0f7bf3d77ddbc2845ab4214f5f55a1608672405284401689f9c8e30844c64a92"} Nov 01 03:53:55.358086 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:55.358081 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg" Nov 01 03:53:55.358587 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:55.358121 6735 scope.go:122] "RemoveContainer" containerID="cc5389bbb70268814a91c2357a3ddcf7ae23459c51b99cb846e6c5bcb0ea1b5e" Nov 01 03:53:55.513080 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:55.513030 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg"] Nov 01 03:53:55.558976 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:55.558947 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c476fff-555db5c8f7-qtqzg"] Nov 01 03:53:56.530168 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:56.530132 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d29490-451d-4636-ac7d-bfd4e1a0f301" path="/var/lib/kubelet/pods/12d29490-451d-4636-ac7d-bfd4e1a0f301/volumes" Nov 01 03:53:56.749489 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:56.749452 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2"] Nov 01 03:53:56.749729 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:56.749702 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" podUID="48318085-4510-4ad4-81fa-8f0fa77f4029" containerName="maas-api" containerID="cri-o://94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2" gracePeriod=30 Nov 01 03:53:57.031325 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.031295 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:57.141105 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.139333 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-metrics-certs\") pod \"48318085-4510-4ad4-81fa-8f0fa77f4029\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " Nov 01 03:53:57.141105 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.139461 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48318085-4510-4ad4-81fa-8f0fa77f4029-kube-api-access-4bz2k\" (UniqueName: \"kubernetes.io/projected/48318085-4510-4ad4-81fa-8f0fa77f4029-kube-api-access-4bz2k\") pod \"48318085-4510-4ad4-81fa-8f0fa77f4029\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " Nov 01 03:53:57.141105 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.139555 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-maas-api-tls\") pod \"48318085-4510-4ad4-81fa-8f0fa77f4029\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " Nov 01 03:53:57.141105 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.139611 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48318085-4510-4ad4-81fa-8f0fa77f4029-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48318085-4510-4ad4-81fa-8f0fa77f4029-openshift-service-ca\") pod \"48318085-4510-4ad4-81fa-8f0fa77f4029\" (UID: \"48318085-4510-4ad4-81fa-8f0fa77f4029\") " Nov 01 03:53:57.147315 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.146993 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48318085-4510-4ad4-81fa-8f0fa77f4029-openshift-service-ca" pod "48318085-4510-4ad4-81fa-8f0fa77f4029" (UID: "48318085-4510-4ad4-81fa-8f0fa77f4029"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:53:57.151773 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.151288 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-maas-api-tls" pod "48318085-4510-4ad4-81fa-8f0fa77f4029" (UID: "48318085-4510-4ad4-81fa-8f0fa77f4029"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:53:57.163154 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.163107 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-metrics-certs" pod "48318085-4510-4ad4-81fa-8f0fa77f4029" (UID: "48318085-4510-4ad4-81fa-8f0fa77f4029"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 03:53:57.163656 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.163628 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48318085-4510-4ad4-81fa-8f0fa77f4029-kube-api-access-4bz2k" pod "48318085-4510-4ad4-81fa-8f0fa77f4029" (UID: "48318085-4510-4ad4-81fa-8f0fa77f4029"). InnerVolumeSpecName "kube-api-access-4bz2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:53:57.241319 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.241289 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:57.241319 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.241314 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48318085-4510-4ad4-81fa-8f0fa77f4029-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:57.241319 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.241323 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48318085-4510-4ad4-81fa-8f0fa77f4029-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:57.241545 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.241333 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bz2k\" (UniqueName: \"kubernetes.io/projected/48318085-4510-4ad4-81fa-8f0fa77f4029-kube-api-access-4bz2k\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:53:57.368643 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.368612 6735 generic.go:356] "Generic (PLEG): container finished" podID="48318085-4510-4ad4-81fa-8f0fa77f4029" containerID="94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2" exitCode=0 Nov 01 03:53:57.368833 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.368661 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" Nov 01 03:53:57.368833 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.368687 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" event={"ID":"48318085-4510-4ad4-81fa-8f0fa77f4029","Type":"ContainerDied","Data":"94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2"} Nov 01 03:53:57.368833 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.368747 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2" event={"ID":"48318085-4510-4ad4-81fa-8f0fa77f4029","Type":"ContainerDied","Data":"910dea5660a78cb91c7bad06b088c4a831ff43c378df36c248832c17489d63cb"} Nov 01 03:53:57.368833 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.368780 6735 scope.go:122] "RemoveContainer" containerID="94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2" Nov 01 03:53:57.377820 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.377804 6735 scope.go:122] "RemoveContainer" containerID="94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2" Nov 01 03:53:57.378084 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:53:57.378065 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2\": container with ID starting with 94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2 not found: ID does not exist" containerID="94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2" Nov 01 03:53:57.378133 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.378094 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2"} err="failed to get container status \"94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2\": rpc error: code = NotFound desc = could not find container \"94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2\": container with ID starting with 94c63526659f49b51d4130385a0c410ed2b2e44fee0c46a630a565e637299fb2 not found: ID does not exist" Nov 01 03:53:57.522989 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.522958 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2"] Nov 01 03:53:57.526990 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.526967 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f50c3be9-79c56cd8bf-dz6t2"] Nov 01 03:53:57.936975 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.936899 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf"] Nov 01 03:53:57.937359 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:57.937191 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" containerID="cri-o://c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08" gracePeriod=30 Nov 01 03:53:58.361769 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:58.361736 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4"] Nov 01 03:53:58.362005 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:58.361982 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" containerID="cri-o://0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859" gracePeriod=30 Nov 01 03:53:58.524404 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:58.524363 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48318085-4510-4ad4-81fa-8f0fa77f4029" path="/var/lib/kubelet/pods/48318085-4510-4ad4-81fa-8f0fa77f4029/volumes" Nov 01 03:53:59.295603 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.295566 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr"] Nov 01 03:53:59.296208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296189 6735 cpu_manager.go:414] "containerMap: removing container" podUID="3018d527-9060-4621-a640-39f5a8fc9ec8" containerName="payload-processing" Nov 01 03:53:59.296286 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296209 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3018d527-9060-4621-a640-39f5a8fc9ec8" containerName="payload-processing" Nov 01 03:53:59.296286 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296228 6735 cpu_manager.go:414] "containerMap: removing container" podUID="48318085-4510-4ad4-81fa-8f0fa77f4029" containerName="maas-api" Nov 01 03:53:59.296286 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296237 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48318085-4510-4ad4-81fa-8f0fa77f4029" containerName="maas-api" Nov 01 03:53:59.296286 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296284 6735 cpu_manager.go:414] "containerMap: removing container" podUID="12d29490-451d-4636-ac7d-bfd4e1a0f301" containerName="maas-api" Nov 01 03:53:59.296414 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296292 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d29490-451d-4636-ac7d-bfd4e1a0f301" containerName="maas-api" Nov 01 03:53:59.296414 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296388 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="3018d527-9060-4621-a640-39f5a8fc9ec8" containerName="payload-processing" Nov 01 03:53:59.296414 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296402 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="48318085-4510-4ad4-81fa-8f0fa77f4029" containerName="maas-api" Nov 01 03:53:59.296500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.296419 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d29490-451d-4636-ac7d-bfd4e1a0f301" containerName="maas-api" Nov 01 03:53:59.301465 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.301443 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.339768 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.339741 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt"] Nov 01 03:53:59.354447 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.354403 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a9c5b3-openshift-default-dockercfg-88kq5\"" Nov 01 03:53:59.373482 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.373458 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr"] Nov 01 03:53:59.409952 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.409927 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh"] Nov 01 03:53:59.467456 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467424 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467627 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467491 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8b0b4-6032-451e-9cc8-0e666197d896-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467627 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467602 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467722 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467635 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467722 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467660 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467722 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467698 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467830 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467739 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467830 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467819 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.467926 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.467858 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc6sh\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-kube-api-access-fc6sh\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568742 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568659 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568742 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568708 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8b0b4-6032-451e-9cc8-0e666197d896-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568753 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568774 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568791 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568815 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568841 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568875 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.568971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.568910 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc6sh\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-kube-api-access-fc6sh\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.569388 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.569139 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.569388 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.569173 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.569388 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.569256 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.569550 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.569391 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.569601 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.569550 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8b0b4-6032-451e-9cc8-0e666197d896-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.571790 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.571766 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.571959 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.571940 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.627215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.627174 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc6sh\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-kube-api-access-fc6sh\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.629690 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.629663 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:53:59.916769 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:53:59.916685 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:00.360193 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.360117 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr"] Nov 01 03:54:00.362627 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:00.362599 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc8b0b4_6032_451e_9cc8_0e666197d896.slice/crio-4838015b3e0c599fcd17eff2f892259a37c5e22182188cf36641917a1a09b425 WatchSource:0}: Error finding container 4838015b3e0c599fcd17eff2f892259a37c5e22182188cf36641917a1a09b425: Status 404 returned error can't find the container with id 4838015b3e0c599fcd17eff2f892259a37c5e22182188cf36641917a1a09b425 Nov 01 03:54:00.364799 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.364766 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:00.364870 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.364831 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:00.364870 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.364862 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:00.395767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.395731 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" event={"ID":"fcc8b0b4-6032-451e-9cc8-0e666197d896","Type":"ContainerStarted","Data":"4838015b3e0c599fcd17eff2f892259a37c5e22182188cf36641917a1a09b425"} Nov 01 03:54:00.699671 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.699589 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj"] Nov 01 03:54:00.703635 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.703602 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.754424 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.754390 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f73b9f-openshift-default-dockercfg-rtg6h\"" Nov 01 03:54:00.762237 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.762200 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj"] Nov 01 03:54:00.784645 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.784607 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.784816 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.784677 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.784892 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.784869 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.784957 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.784930 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.785012 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.784986 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.785098 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.785073 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.785148 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.785116 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxtl\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-kube-api-access-nfxtl\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.785223 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.785205 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4583095-5413-4a7f-9515-f0cab6c7b374-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.785298 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.785282 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.885845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.885806 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.885845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.885842 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886128 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.885858 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxtl\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-kube-api-access-nfxtl\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886128 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.885968 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4583095-5413-4a7f-9515-f0cab6c7b374-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886128 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886028 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886143 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886187 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886218 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886274 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886511 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886292 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886511 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886402 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886825 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886801 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4583095-5413-4a7f-9515-f0cab6c7b374-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886944 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886879 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.886944 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.886887 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.888355 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.888330 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.888708 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.888688 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.943718 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.943689 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:00.943866 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:00.943773 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxtl\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-kube-api-access-nfxtl\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:01.016425 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.016393 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:01.083938 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.083867 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:54:01.402100 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.402003 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" event={"ID":"fcc8b0b4-6032-451e-9cc8-0e666197d896","Type":"ContainerStarted","Data":"265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e"} Nov 01 03:54:01.458983 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.458956 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj"] Nov 01 03:54:01.461762 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:01.461734 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4583095_5413_4a7f_9515_f0cab6c7b374.slice/crio-843c92e341e0bd9e414da3e407070b6ede7bfb6daa79440e3475552e1de06144 WatchSource:0}: Error finding container 843c92e341e0bd9e414da3e407070b6ede7bfb6daa79440e3475552e1de06144: Status 404 returned error can't find the container with id 843c92e341e0bd9e414da3e407070b6ede7bfb6daa79440e3475552e1de06144 Nov 01 03:54:01.463952 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.463924 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:01.464008 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.463987 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:01.464097 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.464018 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:01.518924 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.518869 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podStartSLOduration=2.518850949 podStartE2EDuration="2.518850949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:01.517635394 +0000 UTC m=+653.106730024" watchObservedRunningTime="2026-11-01 03:54:01.518850949 +0000 UTC m=+653.107945580" Nov 01 03:54:01.917003 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.916970 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:01.922390 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:01.922363 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:02.408428 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:02.408394 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" event={"ID":"e4583095-5413-4a7f-9515-f0cab6c7b374","Type":"ContainerStarted","Data":"333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf"} Nov 01 03:54:02.408843 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:02.408450 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" event={"ID":"e4583095-5413-4a7f-9515-f0cab6c7b374","Type":"ContainerStarted","Data":"843c92e341e0bd9e414da3e407070b6ede7bfb6daa79440e3475552e1de06144"} Nov 01 03:54:02.408843 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:02.408478 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:02.409500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:02.409484 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:02.524925 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:02.524864 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podStartSLOduration=2.524846888 podStartE2EDuration="2.524846888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:02.522388067 +0000 UTC m=+654.111482699" watchObservedRunningTime="2026-11-01 03:54:02.524846888 +0000 UTC m=+654.113941518" Nov 01 03:54:02.695161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:02.695083 6735 patch_prober.go:29] interesting pod/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 03:54:02.695161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:02.695132 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 03:54:03.000134 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:03.000097 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:54:03.017543 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.017518 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:03.022787 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.022764 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:03.201610 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.201547 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:54:03.313379 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313347 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-credential-socket\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313379 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313384 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istiod-ca-cert\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313585 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313454 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-podinfo\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313585 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313477 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-envoy\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313585 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313532 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-token\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-token\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313585 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313572 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-certs\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313766 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313598 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-kube-api-access-w5pmj\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-kube-api-access-w5pmj\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313766 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313649 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-credential-socket" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.313766 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313754 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-data\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313917 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313802 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-socket\") pod \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\" (UID: \"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f\") " Nov 01 03:54:03.313917 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313890 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-certs" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.313917 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.313896 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istiod-ca-cert" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:54:03.314091 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.314057 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-data" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.314346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.314183 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-socket" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.314346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.314281 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.314346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.314295 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.314346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.314304 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.314346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.314314 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.314346 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.314328 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.315894 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.315861 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-envoy" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.316009 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.315906 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-kube-api-access-w5pmj" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "kube-api-access-w5pmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:54:03.316120 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.316055 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-podinfo" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 03:54:03.316483 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.316441 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-token" pod "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" (UID: "bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:54:03.414048 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.414000 6735 generic.go:356] "Generic (PLEG): container finished" podID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerID="c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08" exitCode=0 Nov 01 03:54:03.414487 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.414066 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" Nov 01 03:54:03.414487 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.414066 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" event={"ID":"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f","Type":"ContainerDied","Data":"c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08"} Nov 01 03:54:03.414487 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.414123 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf" event={"ID":"bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f","Type":"ContainerDied","Data":"10119f1557ea6d71e25080b7267e50108f39dd7e01692282b31a97c54451698e"} Nov 01 03:54:03.414487 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.414155 6735 scope.go:122] "RemoveContainer" containerID="c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08" Nov 01 03:54:03.414830 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.414808 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:03.415067 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.415051 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.415153 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.415073 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5pmj\" (UniqueName: \"kubernetes.io/projected/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-kube-api-access-w5pmj\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.415153 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.415084 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.415153 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.415097 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.415794 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.415778 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:03.475374 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.475351 6735 scope.go:122] "RemoveContainer" containerID="504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f" Nov 01 03:54:03.483593 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.483571 6735 scope.go:122] "RemoveContainer" containerID="c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08" Nov 01 03:54:03.483859 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:03.483841 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08\": container with ID starting with c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08 not found: ID does not exist" containerID="c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08" Nov 01 03:54:03.483908 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.483869 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08"} err="failed to get container status \"c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08\": rpc error: code = NotFound desc = could not find container \"c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08\": container with ID starting with c250de6dde1e016553606ee24f132c6d650e6192cca069de72e09d42b4755d08 not found: ID does not exist" Nov 01 03:54:03.483908 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.483892 6735 scope.go:122] "RemoveContainer" containerID="504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f" Nov 01 03:54:03.484222 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:03.484199 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f\": container with ID starting with 504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f not found: ID does not exist" containerID="504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f" Nov 01 03:54:03.484297 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.484230 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f"} err="failed to get container status \"504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f\": rpc error: code = NotFound desc = could not find container \"504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f\": container with ID starting with 504ca0f5b3b45714dedd72ea609e2f84059386060acb6c30672dae033731492f not found: ID does not exist" Nov 01 03:54:03.568291 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.568239 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf"] Nov 01 03:54:03.573176 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.573141 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4c476fff-openshift-default-66776c5dbf-fwwdf"] Nov 01 03:54:03.609949 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.609925 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:54:03.718368 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718337 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-data\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718552 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718398 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-kube-api-access-2gzzv\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-kube-api-access-2gzzv\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718552 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718438 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-certs\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718552 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718487 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-token\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-token\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718712 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718550 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-envoy\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718712 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718596 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-podinfo\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718712 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718673 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istiod-ca-cert\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718846 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718697 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-data" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.718846 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718716 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-socket\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718846 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718754 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-credential-socket\") pod \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\" (UID: \"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2\") " Nov 01 03:54:03.718999 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.718968 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-socket" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.719089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.719065 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istiod-ca-cert" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:54:03.719156 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.719137 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.719222 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.719205 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.719273 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.719226 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-credential-socket" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.719449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.719425 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-certs" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.721113 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.721087 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-podinfo" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 03:54:03.721249 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.721187 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-envoy" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:54:03.721412 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.721393 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-kube-api-access-2gzzv" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "kube-api-access-2gzzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:54:03.721412 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.721400 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-token" pod "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" (UID: "17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:54:03.820135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.820024 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.820135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.820076 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.820135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.820086 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.820135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.820095 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gzzv\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-kube-api-access-2gzzv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.820135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.820104 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.820135 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.820116 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:03.820447 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:03.820139 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:04.340661 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.340614 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" podUID="216f8e02-aa6e-4c7e-8686-fd82abe22e58" containerName="payload-processing" containerID="cri-o://058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7" gracePeriod=25 Nov 01 03:54:04.411157 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.411096 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" podUID="592b33e0-0019-4265-b354-fa4c57e496f6" containerName="payload-pre-processing" containerID="cri-o://cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023" gracePeriod=25 Nov 01 03:54:04.421827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.421795 6735 generic.go:356] "Generic (PLEG): container finished" podID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerID="0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859" exitCode=0 Nov 01 03:54:04.422200 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.421850 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" Nov 01 03:54:04.422200 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.421873 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" event={"ID":"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2","Type":"ContainerDied","Data":"0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859"} Nov 01 03:54:04.422200 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.421925 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4" event={"ID":"17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2","Type":"ContainerDied","Data":"604aa6362923259cae25bff6a22f00f2fd69b0ac1bb408bdcf60caabcd2cea3f"} Nov 01 03:54:04.422200 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.421950 6735 scope.go:122] "RemoveContainer" containerID="0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859" Nov 01 03:54:04.524604 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.524569 6735 scope.go:122] "RemoveContainer" containerID="f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d" Nov 01 03:54:04.529576 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.529538 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" path="/var/lib/kubelet/pods/bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f/volumes" Nov 01 03:54:04.537555 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.537527 6735 scope.go:122] "RemoveContainer" containerID="0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859" Nov 01 03:54:04.537958 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:04.537930 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859\": container with ID starting with 0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859 not found: ID does not exist" containerID="0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859" Nov 01 03:54:04.538080 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.537968 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859"} err="failed to get container status \"0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859\": rpc error: code = NotFound desc = could not find container \"0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859\": container with ID starting with 0a940ec1f2779d4c7ca6a5ef12830a75161ed773c2562e2ddebb7d520ab94859 not found: ID does not exist" Nov 01 03:54:04.538080 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.537987 6735 scope.go:122] "RemoveContainer" containerID="f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d" Nov 01 03:54:04.538323 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:04.538305 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d\": container with ID starting with f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d not found: ID does not exist" containerID="f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d" Nov 01 03:54:04.538399 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.538327 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d"} err="failed to get container status \"f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d\": rpc error: code = NotFound desc = could not find container \"f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d\": container with ID starting with f9e186e08c00cc64f5aa9d055f6a2be74c35586e289c20d834d07a1c5e26047d not found: ID does not exist" Nov 01 03:54:04.595299 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.595212 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4"] Nov 01 03:54:04.640746 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.640718 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3145c214-openshift-default-75d84f979f-tfls4"] Nov 01 03:54:04.681956 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.681921 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:54:04.695974 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.695951 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:54:04.831065 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.831005 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592b33e0-0019-4265-b354-fa4c57e496f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592b33e0-0019-4265-b354-fa4c57e496f6-plugins-config-volume\") pod \"592b33e0-0019-4265-b354-fa4c57e496f6\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " Nov 01 03:54:04.831208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.831081 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216f8e02-aa6e-4c7e-8686-fd82abe22e58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f8e02-aa6e-4c7e-8686-fd82abe22e58-plugins-config-volume\") pod \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " Nov 01 03:54:04.831208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.831104 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216f8e02-aa6e-4c7e-8686-fd82abe22e58-kube-api-access-j48t7\" (UniqueName: \"kubernetes.io/projected/216f8e02-aa6e-4c7e-8686-fd82abe22e58-kube-api-access-j48t7\") pod \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\" (UID: \"216f8e02-aa6e-4c7e-8686-fd82abe22e58\") " Nov 01 03:54:04.831208 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.831141 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592b33e0-0019-4265-b354-fa4c57e496f6-kube-api-access-vtvr5\" (UniqueName: \"kubernetes.io/projected/592b33e0-0019-4265-b354-fa4c57e496f6-kube-api-access-vtvr5\") pod \"592b33e0-0019-4265-b354-fa4c57e496f6\" (UID: \"592b33e0-0019-4265-b354-fa4c57e496f6\") " Nov 01 03:54:04.831485 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.831455 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592b33e0-0019-4265-b354-fa4c57e496f6-plugins-config-volume" pod "592b33e0-0019-4265-b354-fa4c57e496f6" (UID: "592b33e0-0019-4265-b354-fa4c57e496f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:54:04.831541 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.831464 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216f8e02-aa6e-4c7e-8686-fd82abe22e58-plugins-config-volume" pod "216f8e02-aa6e-4c7e-8686-fd82abe22e58" (UID: "216f8e02-aa6e-4c7e-8686-fd82abe22e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:54:04.833405 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.833383 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592b33e0-0019-4265-b354-fa4c57e496f6-kube-api-access-vtvr5" pod "592b33e0-0019-4265-b354-fa4c57e496f6" (UID: "592b33e0-0019-4265-b354-fa4c57e496f6"). InnerVolumeSpecName "kube-api-access-vtvr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:54:04.833550 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.833532 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f8e02-aa6e-4c7e-8686-fd82abe22e58-kube-api-access-j48t7" pod "216f8e02-aa6e-4c7e-8686-fd82abe22e58" (UID: "216f8e02-aa6e-4c7e-8686-fd82abe22e58"). InnerVolumeSpecName "kube-api-access-j48t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:54:04.932146 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.932076 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592b33e0-0019-4265-b354-fa4c57e496f6-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:04.932146 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.932102 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f8e02-aa6e-4c7e-8686-fd82abe22e58-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:04.932146 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.932112 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j48t7\" (UniqueName: \"kubernetes.io/projected/216f8e02-aa6e-4c7e-8686-fd82abe22e58-kube-api-access-j48t7\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:04.932146 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:04.932121 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvr5\" (UniqueName: \"kubernetes.io/projected/592b33e0-0019-4265-b354-fa4c57e496f6-kube-api-access-vtvr5\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:54:05.428010 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.427979 6735 generic.go:356] "Generic (PLEG): container finished" podID="216f8e02-aa6e-4c7e-8686-fd82abe22e58" containerID="058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7" exitCode=0 Nov 01 03:54:05.428450 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.428025 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" Nov 01 03:54:05.428450 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.428067 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" event={"ID":"216f8e02-aa6e-4c7e-8686-fd82abe22e58","Type":"ContainerDied","Data":"058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7"} Nov 01 03:54:05.428450 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.428127 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt" event={"ID":"216f8e02-aa6e-4c7e-8686-fd82abe22e58","Type":"ContainerDied","Data":"2dbbde4e0708e02b0c4cf604ea638542b095b35e35620c19af5a066fe20ceba8"} Nov 01 03:54:05.428450 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.428152 6735 scope.go:122] "RemoveContainer" containerID="058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7" Nov 01 03:54:05.429461 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.429429 6735 generic.go:356] "Generic (PLEG): container finished" podID="592b33e0-0019-4265-b354-fa4c57e496f6" containerID="cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023" exitCode=0 Nov 01 03:54:05.429559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.429470 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" Nov 01 03:54:05.429559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.429489 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" event={"ID":"592b33e0-0019-4265-b354-fa4c57e496f6","Type":"ContainerDied","Data":"cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023"} Nov 01 03:54:05.429559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.429532 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh" event={"ID":"592b33e0-0019-4265-b354-fa4c57e496f6","Type":"ContainerDied","Data":"2246c9fd8fe0320d489774a1ee2c05fb9a5a7b0de786b577069dbd4c9c945929"} Nov 01 03:54:05.437629 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.437609 6735 scope.go:122] "RemoveContainer" containerID="058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7" Nov 01 03:54:05.437957 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:05.437937 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7\": container with ID starting with 058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7 not found: ID does not exist" containerID="058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7" Nov 01 03:54:05.438029 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.437968 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7"} err="failed to get container status \"058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7\": rpc error: code = NotFound desc = could not find container \"058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7\": container with ID starting with 058648ae2eb4a750beb6bf70fb701021a55ba2c510c1c39df819b23067eb68e7 not found: ID does not exist" Nov 01 03:54:05.438029 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.437989 6735 scope.go:122] "RemoveContainer" containerID="cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023" Nov 01 03:54:05.445821 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.445802 6735 scope.go:122] "RemoveContainer" containerID="cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023" Nov 01 03:54:05.446090 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:05.446072 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023\": container with ID starting with cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023 not found: ID does not exist" containerID="cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023" Nov 01 03:54:05.446140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.446096 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023"} err="failed to get container status \"cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023\": rpc error: code = NotFound desc = could not find container \"cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023\": container with ID starting with cd391fcdaa378cca6481d9b02c4bfdebbc406820501900a94ef58d50f6e42023 not found: ID does not exist" Nov 01 03:54:05.583544 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.583509 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh"] Nov 01 03:54:05.629762 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.629727 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-48d2fd87-6f9d558c-8zvvh"] Nov 01 03:54:05.732435 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.732401 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt"] Nov 01 03:54:05.737030 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:05.736995 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-48d2fd87-85bc8788c7-6hsjt"] Nov 01 03:54:06.524291 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:06.524262 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" path="/var/lib/kubelet/pods/17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2/volumes" Nov 01 03:54:06.524650 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:06.524634 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216f8e02-aa6e-4c7e-8686-fd82abe22e58" path="/var/lib/kubelet/pods/216f8e02-aa6e-4c7e-8686-fd82abe22e58/volumes" Nov 01 03:54:06.524990 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:06.524976 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592b33e0-0019-4265-b354-fa4c57e496f6" path="/var/lib/kubelet/pods/592b33e0-0019-4265-b354-fa4c57e496f6/volumes" Nov 01 03:54:08.261749 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.261721 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq"] Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262096 6735 cpu_manager.go:414] "containerMap: removing container" podUID="216f8e02-aa6e-4c7e-8686-fd82abe22e58" containerName="payload-processing" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262109 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216f8e02-aa6e-4c7e-8686-fd82abe22e58" containerName="payload-processing" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262124 6735 cpu_manager.go:414] "containerMap: removing container" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262129 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262136 6735 cpu_manager.go:414] "containerMap: removing container" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262141 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262148 6735 cpu_manager.go:414] "containerMap: removing container" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262153 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262162 6735 cpu_manager.go:414] "containerMap: removing container" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" Nov 01 03:54:08.262167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262167 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262204 6735 cpu_manager.go:414] "containerMap: removing container" podUID="592b33e0-0019-4265-b354-fa4c57e496f6" containerName="payload-pre-processing" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262211 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592b33e0-0019-4265-b354-fa4c57e496f6" containerName="payload-pre-processing" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262263 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="592b33e0-0019-4265-b354-fa4c57e496f6" containerName="payload-pre-processing" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262275 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="216f8e02-aa6e-4c7e-8686-fd82abe22e58" containerName="payload-processing" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262286 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262292 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7ed28a-43ad-4a96-ba1b-5a5b0b2ae42f" containerName="istio-proxy" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262297 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" Nov 01 03:54:08.262463 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.262302 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="17aabe7d-4f58-4f9a-ae2f-7f0e79e57fe2" containerName="istio-proxy" Nov 01 03:54:08.266902 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.266880 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.326377 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.326337 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq"] Nov 01 03:54:08.370235 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.370207 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-a9c5b3\"" Nov 01 03:54:08.370384 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.370291 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a9c5b3-dockercfg-c6mp5\"" Nov 01 03:54:08.371690 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.371669 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca23905-14e8-4101-903b-bf0631968c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.371802 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.371789 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn65l\" (UniqueName: \"kubernetes.io/projected/4ca23905-14e8-4101-903b-bf0631968c1f-kube-api-access-jn65l\") pod \"payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.421669 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.421637 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg"] Nov 01 03:54:08.425099 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.425079 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.472565 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.472535 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca23905-14e8-4101-903b-bf0631968c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.472715 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.472635 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn65l\" (UniqueName: \"kubernetes.io/projected/4ca23905-14e8-4101-903b-bf0631968c1f-kube-api-access-jn65l\") pod \"payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.472715 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.472661 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nrq\" (UniqueName: \"kubernetes.io/projected/73b88c83-a504-44d4-8748-6ecb3200bbe1-kube-api-access-r8nrq\") pod \"payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.472715 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.472686 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b88c83-a504-44d4-8748-6ecb3200bbe1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.473268 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.473247 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca23905-14e8-4101-903b-bf0631968c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.483973 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.483939 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg"] Nov 01 03:54:08.529935 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.529872 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn65l\" (UniqueName: \"kubernetes.io/projected/4ca23905-14e8-4101-903b-bf0631968c1f-kube-api-access-jn65l\") pod \"payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.573971 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.573939 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nrq\" (UniqueName: \"kubernetes.io/projected/73b88c83-a504-44d4-8748-6ecb3200bbe1-kube-api-access-r8nrq\") pod \"payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.574176 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.573977 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b88c83-a504-44d4-8748-6ecb3200bbe1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.574582 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.574552 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b88c83-a504-44d4-8748-6ecb3200bbe1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.627008 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.626984 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a9c5b3-dockercfg-c6mp5\"" Nov 01 03:54:08.629179 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.629165 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:08.631380 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.631356 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nrq\" (UniqueName: \"kubernetes.io/projected/73b88c83-a504-44d4-8748-6ecb3200bbe1-kube-api-access-r8nrq\") pod \"payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.738086 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.738049 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:08.993228 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:08.993201 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq"] Nov 01 03:54:08.996047 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:08.995992 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca23905_14e8_4101_903b_bf0631968c1f.slice/crio-7ce3067a74c84e3ff27856c8c5bd99cc9fb2d3ff5bfc1e8226de3f7e143fb11b WatchSource:0}: Error finding container 7ce3067a74c84e3ff27856c8c5bd99cc9fb2d3ff5bfc1e8226de3f7e143fb11b: Status 404 returned error can't find the container with id 7ce3067a74c84e3ff27856c8c5bd99cc9fb2d3ff5bfc1e8226de3f7e143fb11b Nov 01 03:54:09.105394 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.105217 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg"] Nov 01 03:54:09.107407 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:09.107377 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b88c83_a504_44d4_8748_6ecb3200bbe1.slice/crio-9f0c507dfcab841edd8b963c2d4aa54eeddd4c434b54339c28eab56eea8d09b4 WatchSource:0}: Error finding container 9f0c507dfcab841edd8b963c2d4aa54eeddd4c434b54339c28eab56eea8d09b4: Status 404 returned error can't find the container with id 9f0c507dfcab841edd8b963c2d4aa54eeddd4c434b54339c28eab56eea8d09b4 Nov 01 03:54:09.457279 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.457176 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" event={"ID":"4ca23905-14e8-4101-903b-bf0631968c1f","Type":"ContainerStarted","Data":"24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce"} Nov 01 03:54:09.457279 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.457269 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" event={"ID":"4ca23905-14e8-4101-903b-bf0631968c1f","Type":"ContainerStarted","Data":"7ce3067a74c84e3ff27856c8c5bd99cc9fb2d3ff5bfc1e8226de3f7e143fb11b"} Nov 01 03:54:09.457756 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.457302 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:09.458752 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.458730 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" event={"ID":"73b88c83-a504-44d4-8748-6ecb3200bbe1","Type":"ContainerStarted","Data":"67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18"} Nov 01 03:54:09.458859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.458778 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:09.458859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.458796 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" event={"ID":"73b88c83-a504-44d4-8748-6ecb3200bbe1","Type":"ContainerStarted","Data":"9f0c507dfcab841edd8b963c2d4aa54eeddd4c434b54339c28eab56eea8d09b4"} Nov 01 03:54:09.567476 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.567429 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" podStartSLOduration=1.567415203 podStartE2EDuration="1.567415203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:09.566057159 +0000 UTC m=+661.155151779" watchObservedRunningTime="2026-11-01 03:54:09.567415203 +0000 UTC m=+661.156509832" Nov 01 03:54:09.685774 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:09.685229 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" podStartSLOduration=1.685206408 podStartE2EDuration="1.685206408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:09.678906535 +0000 UTC m=+661.268001165" watchObservedRunningTime="2026-11-01 03:54:09.685206408 +0000 UTC m=+661.274301038" Nov 01 03:54:12.814987 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.814948 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds"] Nov 01 03:54:12.818525 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.818503 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:12.877636 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.867863 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-162191\"" Nov 01 03:54:12.915013 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.914977 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-162191-dockercfg-vffzs\"" Nov 01 03:54:12.915664 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.915630 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0baee5-5558-4344-ada8-cc90b190bcfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:12.915889 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.915867 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbpws\" (UniqueName: \"kubernetes.io/projected/6a0baee5-5558-4344-ada8-cc90b190bcfa-kube-api-access-vbpws\") pod \"payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:12.917074 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.916941 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds"] Nov 01 03:54:12.971969 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.971941 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4"] Nov 01 03:54:12.976968 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:12.976940 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.016962 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.016935 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbpws\" (UniqueName: \"kubernetes.io/projected/6a0baee5-5558-4344-ada8-cc90b190bcfa-kube-api-access-vbpws\") pod \"payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:13.017136 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.016983 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2nt7\" (UniqueName: \"kubernetes.io/projected/37b0d047-59ea-4f3e-9d5e-27203656a227-kube-api-access-r2nt7\") pod \"payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.017136 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.017013 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0baee5-5558-4344-ada8-cc90b190bcfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:13.017259 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.017172 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b0d047-59ea-4f3e-9d5e-27203656a227-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.017762 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.017738 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0baee5-5558-4344-ada8-cc90b190bcfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:13.031134 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.031112 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4"] Nov 01 03:54:13.075081 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.075003 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbpws\" (UniqueName: \"kubernetes.io/projected/6a0baee5-5558-4344-ada8-cc90b190bcfa-kube-api-access-vbpws\") pod \"payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:13.118280 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.118255 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b0d047-59ea-4f3e-9d5e-27203656a227-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.118406 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.118323 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2nt7\" (UniqueName: \"kubernetes.io/projected/37b0d047-59ea-4f3e-9d5e-27203656a227-kube-api-access-r2nt7\") pod \"payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.118922 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.118903 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b0d047-59ea-4f3e-9d5e-27203656a227-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.131159 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.131141 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:13.176392 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.176363 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2nt7\" (UniqueName: \"kubernetes.io/projected/37b0d047-59ea-4f3e-9d5e-27203656a227-kube-api-access-r2nt7\") pod \"payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.291579 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.291552 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:13.500521 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.500499 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds"] Nov 01 03:54:13.502869 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:13.502845 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0baee5_5558_4344_ada8_cc90b190bcfa.slice/crio-67e6911abb05be137047e4623faeeae71c19e257bbdf36be73743c4c5e6ea11c WatchSource:0}: Error finding container 67e6911abb05be137047e4623faeeae71c19e257bbdf36be73743c4c5e6ea11c: Status 404 returned error can't find the container with id 67e6911abb05be137047e4623faeeae71c19e257bbdf36be73743c4c5e6ea11c Nov 01 03:54:13.658433 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:13.658403 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4"] Nov 01 03:54:13.660990 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:13.660957 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b0d047_59ea_4f3e_9d5e_27203656a227.slice/crio-12c97bf2bba0443f3368047f067e9280839d51df116f3581c8b1efbd04a84606 WatchSource:0}: Error finding container 12c97bf2bba0443f3368047f067e9280839d51df116f3581c8b1efbd04a84606: Status 404 returned error can't find the container with id 12c97bf2bba0443f3368047f067e9280839d51df116f3581c8b1efbd04a84606 Nov 01 03:54:14.482749 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.482712 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" event={"ID":"37b0d047-59ea-4f3e-9d5e-27203656a227","Type":"ContainerStarted","Data":"d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145"} Nov 01 03:54:14.483248 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.482784 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:14.483248 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.482811 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" event={"ID":"37b0d047-59ea-4f3e-9d5e-27203656a227","Type":"ContainerStarted","Data":"12c97bf2bba0443f3368047f067e9280839d51df116f3581c8b1efbd04a84606"} Nov 01 03:54:14.484280 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.484255 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" event={"ID":"6a0baee5-5558-4344-ada8-cc90b190bcfa","Type":"ContainerStarted","Data":"5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5"} Nov 01 03:54:14.484364 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.484297 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" event={"ID":"6a0baee5-5558-4344-ada8-cc90b190bcfa","Type":"ContainerStarted","Data":"67e6911abb05be137047e4623faeeae71c19e257bbdf36be73743c4c5e6ea11c"} Nov 01 03:54:14.484416 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.484400 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:14.607491 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.607428 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" podStartSLOduration=2.607407887 podStartE2EDuration="2.607407887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:14.605333971 +0000 UTC m=+666.194428603" watchObservedRunningTime="2026-11-01 03:54:14.607407887 +0000 UTC m=+666.196502517" Nov 01 03:54:14.715644 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:14.715584 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" podStartSLOduration=2.715571448 podStartE2EDuration="2.715571448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:14.713907437 +0000 UTC m=+666.303002066" watchObservedRunningTime="2026-11-01 03:54:14.715571448 +0000 UTC m=+666.304666067" Nov 01 03:54:17.449290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.449255 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p"] Nov 01 03:54:17.452740 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.452723 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.555974 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.555940 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f73b9f\"" Nov 01 03:54:17.556226 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.555961 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f73b9f-dockercfg-vw2hn\"" Nov 01 03:54:17.557855 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.557829 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p"] Nov 01 03:54:17.560462 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.560436 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8897b0-1a82-4173-93e8-948fbffe7cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.560578 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.560492 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62mpr\" (UniqueName: \"kubernetes.io/projected/cc8897b0-1a82-4173-93e8-948fbffe7cf6-kube-api-access-62mpr\") pod \"payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.611886 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.611854 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t"] Nov 01 03:54:17.615167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.615148 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:17.661367 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.661339 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8897b0-1a82-4173-93e8-948fbffe7cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.661493 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.661389 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62mpr\" (UniqueName: \"kubernetes.io/projected/cc8897b0-1a82-4173-93e8-948fbffe7cf6-kube-api-access-62mpr\") pod \"payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.661493 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.661461 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:17.661493 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.661478 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6wq\" (UniqueName: \"kubernetes.io/projected/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-kube-api-access-tf6wq\") pod \"payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:17.662086 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.662066 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8897b0-1a82-4173-93e8-948fbffe7cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.672937 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.672913 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t"] Nov 01 03:54:17.721378 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.721332 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62mpr\" (UniqueName: \"kubernetes.io/projected/cc8897b0-1a82-4173-93e8-948fbffe7cf6-kube-api-access-62mpr\") pod \"payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.762825 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.762792 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:17.762991 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.762836 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6wq\" (UniqueName: \"kubernetes.io/projected/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-kube-api-access-tf6wq\") pod \"payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:17.763490 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.763470 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:17.765615 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.765598 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:17.824526 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.822059 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6wq\" (UniqueName: \"kubernetes.io/projected/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-kube-api-access-tf6wq\") pod \"payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:17.928244 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:17.928209 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:18.135976 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.135946 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p"] Nov 01 03:54:18.138523 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:18.138493 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8897b0_1a82_4173_93e8_948fbffe7cf6.slice/crio-ca6a48bf6c0eed09d4e0ba67884d581781d19b8a96e2aa590b424db3ecbf50b5 WatchSource:0}: Error finding container ca6a48bf6c0eed09d4e0ba67884d581781d19b8a96e2aa590b424db3ecbf50b5: Status 404 returned error can't find the container with id ca6a48bf6c0eed09d4e0ba67884d581781d19b8a96e2aa590b424db3ecbf50b5 Nov 01 03:54:18.317635 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.317607 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t"] Nov 01 03:54:18.319630 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:18.319601 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef761b2_65e2_4c8d_bdb7_13004e3d26c0.slice/crio-e91ba2fe63ed7c27cd5c0e6ab11100f0bac6a6a50d0184eabdedf0cae1e42eef WatchSource:0}: Error finding container e91ba2fe63ed7c27cd5c0e6ab11100f0bac6a6a50d0184eabdedf0cae1e42eef: Status 404 returned error can't find the container with id e91ba2fe63ed7c27cd5c0e6ab11100f0bac6a6a50d0184eabdedf0cae1e42eef Nov 01 03:54:18.528099 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.528074 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:18.528442 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.528117 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:18.528442 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.528133 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" event={"ID":"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0","Type":"ContainerStarted","Data":"5123ac7d0b17ada5cb98fcc5cd757091ec17f7b6d1156cda238f2305de0257f9"} Nov 01 03:54:18.528442 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.528150 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" event={"ID":"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0","Type":"ContainerStarted","Data":"e91ba2fe63ed7c27cd5c0e6ab11100f0bac6a6a50d0184eabdedf0cae1e42eef"} Nov 01 03:54:18.528442 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.528165 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" event={"ID":"cc8897b0-1a82-4173-93e8-948fbffe7cf6","Type":"ContainerStarted","Data":"fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077"} Nov 01 03:54:18.528442 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.528178 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" event={"ID":"cc8897b0-1a82-4173-93e8-948fbffe7cf6","Type":"ContainerStarted","Data":"ca6a48bf6c0eed09d4e0ba67884d581781d19b8a96e2aa590b424db3ecbf50b5"} Nov 01 03:54:18.633992 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.633881 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" podStartSLOduration=1.633865858 podStartE2EDuration="1.633865858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:18.63333128 +0000 UTC m=+670.222425909" watchObservedRunningTime="2026-11-01 03:54:18.633865858 +0000 UTC m=+670.222960486" Nov 01 03:54:18.744201 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:18.743230 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" podStartSLOduration=1.743213011 podStartE2EDuration="1.743213011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:18.741977855 +0000 UTC m=+670.331072477" watchObservedRunningTime="2026-11-01 03:54:18.743213011 +0000 UTC m=+670.332307641" Nov 01 03:54:20.465837 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:20.465809 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 03:54:20.466335 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:20.465885 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 03:54:20.766503 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:20.766420 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:54:21.527215 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.527177 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49"] Nov 01 03:54:21.531019 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.530997 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.579931 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.579902 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-162191-serving-cert\"" Nov 01 03:54:21.599546 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.599519 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kv75\" (UniqueName: \"kubernetes.io/projected/a957d6e5-260b-42fc-a274-4e564503c3ab-kube-api-access-6kv75\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.599717 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.599553 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a957d6e5-260b-42fc-a274-4e564503c3ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.599783 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.599752 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-metrics-certs\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.599845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.599808 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.634408 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.628470 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49"] Nov 01 03:54:21.700519 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.700490 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-metrics-certs\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.700519 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.700531 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.700727 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:21.700621 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-serving-cert: secret "maas-api-e2e-worker-w0-162191-serving-cert" not found Nov 01 03:54:21.700727 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:21.700675 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls podName:a957d6e5-260b-42fc-a274-4e564503c3ab nodeName:}" failed. No retries permitted until 2026-11-01 03:54:22.20066042 +0000 UTC m=+673.789755027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls") pod "maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" (UID: "a957d6e5-260b-42fc-a274-4e564503c3ab") : secret "maas-api-e2e-worker-w0-162191-serving-cert" not found Nov 01 03:54:21.700727 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.700709 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kv75\" (UniqueName: \"kubernetes.io/projected/a957d6e5-260b-42fc-a274-4e564503c3ab-kube-api-access-6kv75\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.700836 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.700744 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a957d6e5-260b-42fc-a274-4e564503c3ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.701389 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.701361 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a957d6e5-260b-42fc-a274-4e564503c3ab-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.703076 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.703057 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-metrics-certs\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:21.758619 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:21.758571 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kv75\" (UniqueName: \"kubernetes.io/projected/a957d6e5-260b-42fc-a274-4e564503c3ab-kube-api-access-6kv75\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:22.205736 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:22.205691 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:22.208268 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:22.208235 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls\") pod \"maas-api-e2e-worker-w0-162191-7c7c58645-pcl49\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:22.452326 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:22.452295 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:22.822653 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:22.822631 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49"] Nov 01 03:54:23.558565 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:23.558526 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" event={"ID":"a957d6e5-260b-42fc-a274-4e564503c3ab","Type":"ContainerStarted","Data":"28f109f90dc0b6a0ed68f101c31dbcf3e761c1ee107b7da0e2e05c4a5c32848a"} Nov 01 03:54:24.565445 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:24.565409 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" event={"ID":"a957d6e5-260b-42fc-a274-4e564503c3ab","Type":"ContainerStarted","Data":"d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c"} Nov 01 03:54:24.565857 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:24.565498 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:24.677626 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:24.677574 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" podStartSLOduration=2.6272243680000003 podStartE2EDuration="3.677557339s" totalImagesPullingTime="1.050332971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:24.676439954 +0000 UTC m=+676.265534584" watchObservedRunningTime="2026-11-01 03:54:24.677557339 +0000 UTC m=+676.266652165" Nov 01 03:54:25.491764 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:25.491735 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 03:54:25.491964 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:25.491806 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 03:54:29.533761 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:29.533735 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 03:54:30.538509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:30.538479 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 03:54:30.577123 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:30.577098 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 03:54:31.073808 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.073780 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb"] Nov 01 03:54:31.078202 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.078183 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.106446 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.106420 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18525227-695f-424a-80dd-9b2e56509cb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.106584 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.106452 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45kt\" (UniqueName: \"kubernetes.io/projected/18525227-695f-424a-80dd-9b2e56509cb4-kube-api-access-w45kt\") pod \"payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.127455 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.127428 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-865066\"" Nov 01 03:54:31.127455 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.127438 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-865066-dockercfg-qr8g4\"" Nov 01 03:54:31.132539 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.132516 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb"] Nov 01 03:54:31.208124 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.208070 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18525227-695f-424a-80dd-9b2e56509cb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.208308 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.208128 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45kt\" (UniqueName: \"kubernetes.io/projected/18525227-695f-424a-80dd-9b2e56509cb4-kube-api-access-w45kt\") pod \"payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.209021 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.208993 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18525227-695f-424a-80dd-9b2e56509cb4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.272274 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.272235 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45kt\" (UniqueName: \"kubernetes.io/projected/18525227-695f-424a-80dd-9b2e56509cb4-kube-api-access-w45kt\") pod \"payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.391724 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.391632 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:31.784316 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:31.784286 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb"] Nov 01 03:54:31.786343 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:31.786318 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18525227_695f_424a_80dd_9b2e56509cb4.slice/crio-f19ce1d4676a52d644f01c578c7d748addc9c3f2284a386895bc6ba11d004340 WatchSource:0}: Error finding container f19ce1d4676a52d644f01c578c7d748addc9c3f2284a386895bc6ba11d004340: Status 404 returned error can't find the container with id f19ce1d4676a52d644f01c578c7d748addc9c3f2284a386895bc6ba11d004340 Nov 01 03:54:32.612795 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:32.612760 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" event={"ID":"18525227-695f-424a-80dd-9b2e56509cb4","Type":"ContainerStarted","Data":"b2ac37304f78db1f5bae6d56173f8bb36dc13e8fb36c32c4726045937ad3a300"} Nov 01 03:54:32.612961 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:32.612819 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:32.612961 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:32.612835 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" event={"ID":"18525227-695f-424a-80dd-9b2e56509cb4","Type":"ContainerStarted","Data":"f19ce1d4676a52d644f01c578c7d748addc9c3f2284a386895bc6ba11d004340"} Nov 01 03:54:32.741075 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:32.741002 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" podStartSLOduration=1.740987492 podStartE2EDuration="1.740987492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:32.738519921 +0000 UTC m=+684.327614551" watchObservedRunningTime="2026-11-01 03:54:32.740987492 +0000 UTC m=+684.330082139" Nov 01 03:54:33.000852 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:54:33.000813 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:54:38.646299 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:38.646267 6735 generic.go:356] "Generic (PLEG): container finished" podID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerID="265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e" exitCode=0 Nov 01 03:54:38.646772 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:38.646347 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" event={"ID":"fcc8b0b4-6032-451e-9cc8-0e666197d896","Type":"ContainerDied","Data":"265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e"} Nov 01 03:54:38.646772 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:38.646719 6735 scope.go:122] "RemoveContainer" containerID="265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e" Nov 01 03:54:38.648048 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:38.648006 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:38.648160 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:38.648085 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:38.648160 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:38.648119 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:38.973687 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:38.973650 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:54:39.670062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:39.669840 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" event={"ID":"fcc8b0b4-6032-451e-9cc8-0e666197d896","Type":"ContainerStarted","Data":"d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011"} Nov 01 03:54:39.916929 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:39.916881 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:39.917394 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:39.917371 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 03:54:39.917483 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:39.917459 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 03:54:40.917615 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:40.917157 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 03:54:40.917615 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:40.917216 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 03:54:41.917669 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:41.917632 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 03:54:41.918107 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:41.917710 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 03:54:42.917559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:42.917522 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 03:54:42.917765 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:42.917586 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 03:54:43.620217 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:43.620193 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 03:54:43.917615 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:43.917529 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 03:54:43.917615 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:43.917582 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 03:54:45.171704 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:45.171505 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:45.172264 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:45.172239 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:45.173151 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:45.173098 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 03:54:46.652842 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:46.652809 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:54:48.415448 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.415417 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 03:54:48.415865 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.415472 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 03:54:48.745221 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.745192 6735 generic.go:356] "Generic (PLEG): container finished" podID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerID="333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf" exitCode=0 Nov 01 03:54:48.745389 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.745266 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" event={"ID":"e4583095-5413-4a7f-9515-f0cab6c7b374","Type":"ContainerDied","Data":"333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf"} Nov 01 03:54:48.745655 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.745632 6735 scope.go:122] "RemoveContainer" containerID="333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf" Nov 01 03:54:48.746859 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.746828 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:48.746955 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.746896 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:48.746955 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:48.746926 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:54:49.773002 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:49.772969 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" event={"ID":"e4583095-5413-4a7f-9515-f0cab6c7b374","Type":"ContainerStarted","Data":"05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990"} Nov 01 03:54:50.016914 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:50.016877 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:50.017122 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:50.017100 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 03:54:50.017181 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:50.017145 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 03:54:51.017164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:51.017132 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 03:54:51.017578 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:51.017190 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 03:54:52.016970 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.016932 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 03:54:52.017171 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.016996 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 03:54:52.776919 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.776881 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv"] Nov 01 03:54:52.780979 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.780957 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:52.889730 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.882166 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv"] Nov 01 03:54:52.889730 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.882296 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c2c259\"" Nov 01 03:54:52.889730 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.882362 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c2c259-dockercfg-nx6dj\"" Nov 01 03:54:52.908475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.908445 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfs2\" (UniqueName: \"kubernetes.io/projected/78b560bf-0e29-416b-9f70-00223c46a8d1-kube-api-access-rgfs2\") pod \"payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:52.908702 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:52.908688 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b560bf-0e29-416b-9f70-00223c46a8d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:53.009390 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.009358 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfs2\" (UniqueName: \"kubernetes.io/projected/78b560bf-0e29-416b-9f70-00223c46a8d1-kube-api-access-rgfs2\") pod \"payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:53.009585 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.009409 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b560bf-0e29-416b-9f70-00223c46a8d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:53.010263 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.010238 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b560bf-0e29-416b-9f70-00223c46a8d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:53.017372 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.017343 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 03:54:53.017471 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.017394 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 03:54:53.073141 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.073014 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfs2\" (UniqueName: \"kubernetes.io/projected/78b560bf-0e29-416b-9f70-00223c46a8d1-kube-api-access-rgfs2\") pod \"payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:53.094381 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.094352 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:53.472946 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.472918 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv"] Nov 01 03:54:53.475687 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:54:53.475623 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b560bf_0e29_416b_9f70_00223c46a8d1.slice/crio-d5dca37892ef4c0515e928fadbdff867ba68f6c9d2ed029256c8deb774bf2d03 WatchSource:0}: Error finding container d5dca37892ef4c0515e928fadbdff867ba68f6c9d2ed029256c8deb774bf2d03: Status 404 returned error can't find the container with id d5dca37892ef4c0515e928fadbdff867ba68f6c9d2ed029256c8deb774bf2d03 Nov 01 03:54:53.795434 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.795334 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" event={"ID":"78b560bf-0e29-416b-9f70-00223c46a8d1","Type":"ContainerStarted","Data":"d222fd530c181797415f7f43122fb9e13027201c7c8bd9c5ba19f1dce90f4a30"} Nov 01 03:54:53.795434 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.795394 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" event={"ID":"78b560bf-0e29-416b-9f70-00223c46a8d1","Type":"ContainerStarted","Data":"d5dca37892ef4c0515e928fadbdff867ba68f6c9d2ed029256c8deb774bf2d03"} Nov 01 03:54:53.795886 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.795528 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:54:53.905307 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:53.905252 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" podStartSLOduration=1.9052358740000002 podStartE2EDuration="1.905235874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:53.904115172 +0000 UTC m=+705.493209802" watchObservedRunningTime="2026-11-01 03:54:53.905235874 +0000 UTC m=+705.494330504" Nov 01 03:54:54.017120 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:54.017082 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 01 03:54:54.017317 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:54.017142 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 01 03:54:55.021226 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:55.021192 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:55.021681 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:55.021647 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:55.022364 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:55.022345 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 03:54:59.596987 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:54:59.596949 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:55:03.001292 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:55:03.001256 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:55:04.803303 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:04.803275 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 03:55:11.138121 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.138088 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn"] Nov 01 03:55:11.142988 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.142971 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.192270 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.192228 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c2c259-serving-cert\"" Nov 01 03:55:11.200266 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.200238 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c5ccae-f7ae-41be-bca6-decf8a9d8481-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.200393 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.200296 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jhw\" (UniqueName: \"kubernetes.io/projected/74c5ccae-f7ae-41be-bca6-decf8a9d8481-kube-api-access-b9jhw\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.200517 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.200493 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-metrics-certs\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.200612 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.200585 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.240140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.240102 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn"] Nov 01 03:55:11.301807 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.301777 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jhw\" (UniqueName: \"kubernetes.io/projected/74c5ccae-f7ae-41be-bca6-decf8a9d8481-kube-api-access-b9jhw\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.302006 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.301901 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-metrics-certs\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.302006 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.301956 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.302006 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.301981 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c5ccae-f7ae-41be-bca6-decf8a9d8481-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.302181 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:55:11.302121 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-serving-cert: secret "maas-api-e2e-worker-w1-c2c259-serving-cert" not found Nov 01 03:55:11.302235 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:55:11.302182 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls podName:74c5ccae-f7ae-41be-bca6-decf8a9d8481 nodeName:}" failed. No retries permitted until 2026-11-01 03:55:11.80216625 +0000 UTC m=+723.391260860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls") pod "maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" (UID: "74c5ccae-f7ae-41be-bca6-decf8a9d8481") : secret "maas-api-e2e-worker-w1-c2c259-serving-cert" not found Nov 01 03:55:11.302684 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.302662 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c5ccae-f7ae-41be-bca6-decf8a9d8481-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.304419 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.304402 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-metrics-certs\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.362907 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.362876 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jhw\" (UniqueName: \"kubernetes.io/projected/74c5ccae-f7ae-41be-bca6-decf8a9d8481-kube-api-access-b9jhw\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.808093 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.808059 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:11.810347 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:11.810322 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:12.055384 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:12.055346 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:12.439454 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:12.439429 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn"] Nov 01 03:55:12.441637 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:55:12.441606 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c5ccae_f7ae_41be_bca6_decf8a9d8481.slice/crio-f36089bbdf4ca9017908abaa35bc2814c5b508be036105bae27338ed1422e8bb WatchSource:0}: Error finding container f36089bbdf4ca9017908abaa35bc2814c5b508be036105bae27338ed1422e8bb: Status 404 returned error can't find the container with id f36089bbdf4ca9017908abaa35bc2814c5b508be036105bae27338ed1422e8bb Nov 01 03:55:12.894271 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:12.894239 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" event={"ID":"74c5ccae-f7ae-41be-bca6-decf8a9d8481","Type":"ContainerStarted","Data":"f36089bbdf4ca9017908abaa35bc2814c5b508be036105bae27338ed1422e8bb"} Nov 01 03:55:13.900640 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:13.900603 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" event={"ID":"74c5ccae-f7ae-41be-bca6-decf8a9d8481","Type":"ContainerStarted","Data":"9921c6abaedc1fbb7e1be9470e9d5a8b51ac0cd265908d202733b41316e10f54"} Nov 01 03:55:13.901097 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:13.900748 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:14.012193 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:14.012091 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" podStartSLOduration=1.735206288 podStartE2EDuration="3.012078321s" totalImagesPullingTime="1.276872033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:55:14.011380479 +0000 UTC m=+725.600475108" watchObservedRunningTime="2026-11-01 03:55:14.012078321 +0000 UTC m=+725.601172950" Nov 01 03:55:19.913936 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:19.913736 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 03:55:20.079500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:20.079471 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:55:30.664503 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:30.664470 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:55:33.000029 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:55:32.999992 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:55:54.800841 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:55:54.800811 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:56:03.001094 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:56:03.001052 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:56:13.798441 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:13.798405 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:56:31.029954 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.029918 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd"] Nov 01 03:56:31.034163 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.034142 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.084167 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.084139 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-162191-openshift-default-dockercfg-c4j58\"" Nov 01 03:56:31.118610 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.118579 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7401861-17b9-48d2-b5bd-705c2368ed67-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.118610 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.118614 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.118805 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.118635 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.118843 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.118808 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.118889 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.118849 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.118969 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.118949 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jntmp\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-kube-api-access-jntmp\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.119027 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.119006 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.119109 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.119064 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.119109 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.119095 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.132400 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.132371 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:56:31.134125 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.134101 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd"] Nov 01 03:56:31.220001 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.219968 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220331 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220065 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220331 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220120 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jntmp\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-kube-api-access-jntmp\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220451 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220537 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220513 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220590 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220562 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220644 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220592 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220644 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220627 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7401861-17b9-48d2-b5bd-705c2368ed67-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220743 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220657 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.220743 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.220687 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.221458 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.221430 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.222077 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.222054 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.222556 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.222529 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7401861-17b9-48d2-b5bd-705c2368ed67-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.231734 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.231661 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.233757 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.233709 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.238960 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.238934 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.280774 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.280703 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.280907 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.280815 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jntmp\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-kube-api-access-jntmp\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.347852 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.347821 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:31.762462 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.762437 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd"] Nov 01 03:56:31.764907 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:56:31.764877 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7401861_17b9_48d2_b5bd_705c2368ed67.slice/crio-c3ee8d0ecc004e8af8b994c5191668e2cad2f480366de501aa29a55550ffe8ec WatchSource:0}: Error finding container c3ee8d0ecc004e8af8b994c5191668e2cad2f480366de501aa29a55550ffe8ec: Status 404 returned error can't find the container with id c3ee8d0ecc004e8af8b994c5191668e2cad2f480366de501aa29a55550ffe8ec Nov 01 03:56:31.767447 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.767409 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:56:31.767551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.767532 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:56:31.767597 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:31.767584 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:56:32.314820 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:32.314786 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" event={"ID":"e7401861-17b9-48d2-b5bd-705c2368ed67","Type":"ContainerStarted","Data":"79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53"} Nov 01 03:56:32.315269 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:32.314844 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" event={"ID":"e7401861-17b9-48d2-b5bd-705c2368ed67","Type":"ContainerStarted","Data":"c3ee8d0ecc004e8af8b994c5191668e2cad2f480366de501aa29a55550ffe8ec"} Nov 01 03:56:32.363103 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:32.355997 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:32.363103 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:32.356176 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 03:56:32.363103 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:32.356233 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 03:56:32.436333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:32.436280 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podStartSLOduration=2.436264997 podStartE2EDuration="2.436264997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:56:32.434476661 +0000 UTC m=+804.023571290" watchObservedRunningTime="2026-11-01 03:56:32.436264997 +0000 UTC m=+804.025359625" Nov 01 03:56:33.002210 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:56:33.002165 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:56:33.370119 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:33.369135 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 03:56:33.370119 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:33.369207 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 03:56:34.366062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:34.359468 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 03:56:34.366062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:34.359535 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 03:56:35.360061 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:35.359430 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 03:56:35.360061 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:35.359496 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 03:56:36.361405 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:36.361366 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 03:56:36.362008 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:36.361979 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 03:56:37.360062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:37.355429 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 01 03:56:37.360062 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:37.355489 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 01 03:56:38.382988 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:38.382945 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:38.383491 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:38.383450 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:38.384227 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:38.384206 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 03:56:49.320551 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:56:49.320517 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:57:03.001296 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:57:03.001259 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:57:09.426583 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:09.426547 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5"] Nov 01 03:57:09.427082 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:09.426743 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" containerID="cri-o://e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf" gracePeriod=30 Nov 01 03:57:14.675807 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.675781 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:57:14.779877 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.779848 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-certs\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.779889 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-kube-api-access-kttxp\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-kube-api-access-kttxp\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.779927 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-socket\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.779959 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-data\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780009 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-credential-socket\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780088 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-token\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-token\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780145 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-podinfo\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780179 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-certs" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:57:14.780293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780202 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59dcca89-f3aa-437f-9f17-f525bd108fb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dcca89-f3aa-437f-9f17-f525bd108fb9-istiod-ca-cert\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780293 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780261 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-envoy\") pod \"59dcca89-f3aa-437f-9f17-f525bd108fb9\" (UID: \"59dcca89-f3aa-437f-9f17-f525bd108fb9\") " Nov 01 03:57:14.780532 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780301 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-data" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:57:14.780532 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780363 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-socket" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:57:14.780532 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780383 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-credential-socket" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:57:14.780674 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780636 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59dcca89-f3aa-437f-9f17-f525bd108fb9-istiod-ca-cert" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:57:14.780762 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780741 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.780820 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780768 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.780820 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780782 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.780820 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780797 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.780820 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.780810 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59dcca89-f3aa-437f-9f17-f525bd108fb9-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.782297 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.782274 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-kube-api-access-kttxp" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "kube-api-access-kttxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:57:14.782454 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.782435 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-podinfo" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 03:57:14.782874 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.782854 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-token" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:57:14.782966 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.782867 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-envoy" pod "59dcca89-f3aa-437f-9f17-f525bd108fb9" (UID: "59dcca89-f3aa-437f-9f17-f525bd108fb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 03:57:14.881905 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.881871 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.881905 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.881900 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.881905 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.881910 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttxp\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-kube-api-access-kttxp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:14.882160 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:14.881921 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59dcca89-f3aa-437f-9f17-f525bd108fb9-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:57:15.532603 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.532567 6735 generic.go:356] "Generic (PLEG): container finished" podID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerID="e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf" exitCode=0 Nov 01 03:57:15.532780 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.532618 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" Nov 01 03:57:15.532780 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.532627 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" event={"ID":"59dcca89-f3aa-437f-9f17-f525bd108fb9","Type":"ContainerDied","Data":"e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf"} Nov 01 03:57:15.532780 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.532679 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5" event={"ID":"59dcca89-f3aa-437f-9f17-f525bd108fb9","Type":"ContainerDied","Data":"22b74eff0efc9891e776d0c7a0dd1b8dc00ba8a96131f1bb8365691d790cbe5d"} Nov 01 03:57:15.532780 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.532709 6735 scope.go:122] "RemoveContainer" containerID="e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf" Nov 01 03:57:15.542644 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.542622 6735 scope.go:122] "RemoveContainer" containerID="e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf" Nov 01 03:57:15.542942 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:57:15.542913 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf\": container with ID starting with e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf not found: ID does not exist" containerID="e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf" Nov 01 03:57:15.543219 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.542947 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf"} err="failed to get container status \"e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf\": rpc error: code = NotFound desc = could not find container \"e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf\": container with ID starting with e3c1a9bcad5e5766abc3d3154b06c945a3ba9c6c0bb5bbbb5db06eab01e655cf not found: ID does not exist" Nov 01 03:57:15.686981 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.686950 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5"] Nov 01 03:57:15.691416 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:15.691391 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n4hh5"] Nov 01 03:57:16.524688 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:16.524659 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" path="/var/lib/kubelet/pods/59dcca89-f3aa-437f-9f17-f525bd108fb9/volumes" Nov 01 03:57:16.539817 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:16.539790 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:57:33.001015 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:57:33.000980 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:57:48.991639 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:57:48.991596 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:58:03.000761 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:58:03.000714 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:58:08.607852 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:08.607823 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:58:08.609475 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:08.609448 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 03:58:12.991573 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:12.991538 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb"] Nov 01 03:58:12.992010 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:12.991986 6735 cpu_manager.go:414] "containerMap: removing container" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" Nov 01 03:58:12.992010 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:12.991997 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" Nov 01 03:58:12.992127 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:12.992085 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="59dcca89-f3aa-437f-9f17-f525bd108fb9" containerName="istio-proxy" Nov 01 03:58:12.996556 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:12.996537 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.045799 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.045771 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c2c259-openshift-default-dockercfg-7n2f9\"" Nov 01 03:58:13.076512 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076481 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.076670 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076538 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.076670 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076628 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.076758 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076670 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.076758 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076702 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.076887 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076863 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.076926 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076909 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.076966 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076936 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxm8q\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-kube-api-access-nxm8q\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.077013 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.076992 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.095011 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.094968 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb"] Nov 01 03:58:13.178200 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178161 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178397 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178212 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178397 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178237 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxm8q\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-kube-api-access-nxm8q\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178397 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178272 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178397 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178343 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178397 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178384 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178707 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178416 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178707 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178451 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178707 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178495 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178707 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178560 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178707 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178607 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178707 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178643 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.178921 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.178868 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.179140 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.179119 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.180772 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.180745 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.181182 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.181164 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.236148 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.236123 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.236268 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.236247 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxm8q\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-kube-api-access-nxm8q\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.309783 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.309708 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:13.723908 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.723885 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb"] Nov 01 03:58:13.726130 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:58:13.726104 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6b4475_8ef9_47fa_a963_e3b2d77b44b1.slice/crio-979156c9a5d9721b0526b291428d37b7d153fef96d0aafdc0012a380cd309a64 WatchSource:0}: Error finding container 979156c9a5d9721b0526b291428d37b7d153fef96d0aafdc0012a380cd309a64: Status 404 returned error can't find the container with id 979156c9a5d9721b0526b291428d37b7d153fef96d0aafdc0012a380cd309a64 Nov 01 03:58:13.728324 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.728293 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:13.728391 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.728358 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:13.728391 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.728387 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:13.825093 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.825053 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" event={"ID":"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1","Type":"ContainerStarted","Data":"f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e"} Nov 01 03:58:13.825265 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.825119 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" event={"ID":"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1","Type":"ContainerStarted","Data":"979156c9a5d9721b0526b291428d37b7d153fef96d0aafdc0012a380cd309a64"} Nov 01 03:58:13.946674 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:13.946624 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podStartSLOduration=1.946609314 podStartE2EDuration="1.946609314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:13.944706851 +0000 UTC m=+905.533801480" watchObservedRunningTime="2026-11-01 03:58:13.946609314 +0000 UTC m=+905.535703944" Nov 01 03:58:14.310927 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:14.310891 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:14.312701 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:14.312676 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 03:58:14.312827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:14.312725 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 03:58:15.310533 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:15.310497 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 03:58:15.310686 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:15.310548 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 03:58:15.910876 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:15.910842 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6"] Nov 01 03:58:15.914632 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:15.914615 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.006845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.006810 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-routing-overlay-volume\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.006998 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.006853 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhc4\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-kube-api-access-zhhc4\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.006998 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.006914 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.007123 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.006995 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-plugins-config-volume\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.010765 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.010744 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 03:58:16.010836 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.010744 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1fec6c1a0815\"" Nov 01 03:58:16.011075 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.011053 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 03:58:16.011174 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.011108 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 03:58:16.011174 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.011119 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-87c6e6db2865\"" Nov 01 03:58:16.011174 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.011122 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 03:58:16.011174 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.011138 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wgxn8\"" Nov 01 03:58:16.012783 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.012766 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6"] Nov 01 03:58:16.107926 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.107894 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-routing-overlay-volume\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.108145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.107939 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhc4\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-kube-api-access-zhhc4\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.108145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.107995 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.108145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.108018 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-plugins-config-volume\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.108693 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.108673 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-routing-overlay-volume\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.108767 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.108748 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-plugins-config-volume\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.110841 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.110817 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.170742 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.170651 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhc4\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-kube-api-access-zhhc4\") pod \"payload-processing-external-model-77d5dcc848-45zm6\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.227689 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.227658 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.312117 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.312073 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 03:58:16.312320 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.312150 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 03:58:16.521071 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.515065 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:58:16.679880 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.679851 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6"] Nov 01 03:58:16.844055 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.843554 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" event={"ID":"34135bd6-9ca0-40ab-8745-f614b2644d39","Type":"ContainerStarted","Data":"ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f"} Nov 01 03:58:16.844055 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.843630 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:16.844055 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.843656 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" event={"ID":"34135bd6-9ca0-40ab-8745-f614b2644d39","Type":"ContainerStarted","Data":"d7aa8ce06fb9dc3f63af2eddb125ddcf9b5925ce1167186c5bff624fda20253e"} Nov 01 03:58:16.960056 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:16.959843 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" podStartSLOduration=1.959821437 podStartE2EDuration="1.959821437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:16.958819773 +0000 UTC m=+908.547914403" watchObservedRunningTime="2026-11-01 03:58:16.959821437 +0000 UTC m=+908.548916067" Nov 01 03:58:17.312922 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:17.312484 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 03:58:17.312922 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:17.312545 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 03:58:18.310956 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:18.310914 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 03:58:18.311387 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:18.310986 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 03:58:19.311074 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:19.311022 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 03:58:19.311493 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:19.311095 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 03:58:20.504746 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:20.504710 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:20.505406 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:20.505169 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:20.506157 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:20.506130 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 03:58:27.867562 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:27.867534 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 03:58:33.000917 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:58:33.000880 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:58:40.045970 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.045935 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d44b46b89-qvj52"] Nov 01 03:58:40.050118 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.050096 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.124907 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.124870 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-plugins-config-volume\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.125112 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.125003 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k464\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-kube-api-access-8k464\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.125236 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.125199 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-provider-credentials\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.125322 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.125305 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-routing-overlay-volume\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.147690 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.147664 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d44b46b89-qvj52"] Nov 01 03:58:40.149162 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.149133 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm"] Nov 01 03:58:40.153417 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.153393 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.219628 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.219490 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm"] Nov 01 03:58:40.230515 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230483 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa25678-12f9-454a-890e-6ca831c89485-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.230695 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230534 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-plugins-config-volume\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.230695 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230628 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.230814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230691 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k464\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-kube-api-access-8k464\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.230814 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230800 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpppp\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-kube-api-access-qpppp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.230917 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230840 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.230969 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230928 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.231022 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.230975 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-provider-credentials\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.231106 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.231064 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.231164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.231125 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa25678-12f9-454a-890e-6ca831c89485-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.231164 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.231156 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-routing-overlay-volume\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.231266 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.231182 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.231266 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.231208 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.231368 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.231341 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-plugins-config-volume\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.231915 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.231889 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-routing-overlay-volume\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.233658 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.233636 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-provider-credentials\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.288775 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.288745 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k464\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-kube-api-access-8k464\") pod \"payload-processing-external-model-d44b46b89-qvj52\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.332213 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332117 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpppp\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-kube-api-access-qpppp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332213 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332175 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332220 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332256 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332302 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa25678-12f9-454a-890e-6ca831c89485-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332328 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332351 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332386 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa25678-12f9-454a-890e-6ca831c89485-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332743 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332460 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332743 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332590 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332743 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332687 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332902 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332805 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.332946 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.332916 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.333086 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.333062 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa25678-12f9-454a-890e-6ca831c89485-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.334827 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.334804 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.335109 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.335094 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa25678-12f9-454a-890e-6ca831c89485-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.364195 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.364169 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:40.390055 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.390017 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpppp\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-kube-api-access-qpppp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.390187 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.390168 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-c5zfm\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.479579 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.479545 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:40.732695 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.732670 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d44b46b89-qvj52"] Nov 01 03:58:40.734788 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:58:40.734754 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2808c7_2ff6_4571_893b_9b705fe01423.slice/crio-d34e45c2fb8f8b658ff030d9d73a815e8ce9138c6b4cf4bef0875cb780ff549c WatchSource:0}: Error finding container d34e45c2fb8f8b658ff030d9d73a815e8ce9138c6b4cf4bef0875cb780ff549c: Status 404 returned error can't find the container with id d34e45c2fb8f8b658ff030d9d73a815e8ce9138c6b4cf4bef0875cb780ff549c Nov 01 03:58:40.852134 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.852110 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm"] Nov 01 03:58:40.853711 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:58:40.853684 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa25678_12f9_454a_890e_6ca831c89485.slice/crio-d7d19f36ebaaa017e5052b86b72153f85ea1f9756f215537c6424a8b5515aa75 WatchSource:0}: Error finding container d7d19f36ebaaa017e5052b86b72153f85ea1f9756f215537c6424a8b5515aa75: Status 404 returned error can't find the container with id d7d19f36ebaaa017e5052b86b72153f85ea1f9756f215537c6424a8b5515aa75 Nov 01 03:58:40.856115 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.856086 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:40.856186 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.856149 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:40.856186 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:40.856176 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:41.001171 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.001071 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" event={"ID":"3aa25678-12f9-454a-890e-6ca831c89485","Type":"ContainerStarted","Data":"2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4"} Nov 01 03:58:41.001171 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.001154 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" event={"ID":"3aa25678-12f9-454a-890e-6ca831c89485","Type":"ContainerStarted","Data":"d7d19f36ebaaa017e5052b86b72153f85ea1f9756f215537c6424a8b5515aa75"} Nov 01 03:58:41.002965 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.002939 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" event={"ID":"aa2808c7-2ff6-4571-893b-9b705fe01423","Type":"ContainerStarted","Data":"cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5"} Nov 01 03:58:41.003119 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.002986 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" event={"ID":"aa2808c7-2ff6-4571-893b-9b705fe01423","Type":"ContainerStarted","Data":"d34e45c2fb8f8b658ff030d9d73a815e8ce9138c6b4cf4bef0875cb780ff549c"} Nov 01 03:58:41.003119 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.003083 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:58:41.116145 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.116085 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podStartSLOduration=1.116065913 podStartE2EDuration="1.116065913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:41.113566911 +0000 UTC m=+932.702661531" watchObservedRunningTime="2026-11-01 03:58:41.116065913 +0000 UTC m=+932.705160543" Nov 01 03:58:41.224745 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.224694 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podStartSLOduration=1.224676747 podStartE2EDuration="1.224676747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:41.222560682 +0000 UTC m=+932.811655323" watchObservedRunningTime="2026-11-01 03:58:41.224676747 +0000 UTC m=+932.813771412" Nov 01 03:58:41.480382 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.480350 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:41.481721 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.481696 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:41.481831 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:41.481741 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:42.480249 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:42.480213 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:42.480642 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:42.480278 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:43.480748 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:43.480706 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:43.481270 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:43.480773 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:44.480566 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:44.480529 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:44.480761 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:44.480581 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:45.480765 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:45.480732 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:45.481173 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:45.480784 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:46.480799 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:46.480762 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:46.481254 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:46.480819 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:47.480312 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:47.480278 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:47.480499 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:47.480331 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:48.480151 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:48.480121 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 03:58:48.480540 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:48.480165 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 03:58:49.484693 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:49.484666 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:49.485209 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:49.485086 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:49.485891 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:49.485865 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 03:58:49.691929 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:49.691895 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:58:52.012189 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:52.012146 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 03:58:55.254849 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.254815 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm"] Nov 01 03:58:55.258909 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.258890 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.301353 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.301324 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm"] Nov 01 03:58:55.305000 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.304984 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.390254 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.390216 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm"] Nov 01 03:58:55.423449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423416 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76z5\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-kube-api-access-j76z5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.423449 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423451 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0058a65d-d7aa-4969-9c60-e103cef65665-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.423673 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423476 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.423673 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423548 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.423673 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423624 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.423673 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423663 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.423870 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423681 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.423870 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423754 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.423870 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423803 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.423979 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423897 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cb51f3-9162-4372-b68e-5092fc67600e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.423979 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423927 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tdq\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-kube-api-access-p2tdq\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.423979 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.423946 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.424134 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.424046 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.424178 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.424130 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.424230 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.424193 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.424230 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.424211 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0058a65d-d7aa-4969-9c60-e103cef65665-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.424230 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.424227 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.424338 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.424273 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cb51f3-9162-4372-b68e-5092fc67600e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.448210 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.448182 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm"] Nov 01 03:58:55.525500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525419 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.525500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525453 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.525500 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525470 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525510 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525539 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525569 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525595 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525647 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cb51f3-9162-4372-b68e-5092fc67600e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525670 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tdq\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-kube-api-access-p2tdq\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525697 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525725 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.525785 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525768 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525829 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525856 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0058a65d-d7aa-4969-9c60-e103cef65665-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525879 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525910 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cb51f3-9162-4372-b68e-5092fc67600e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525986 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76z5\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-kube-api-access-j76z5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.525990 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.526017 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0058a65d-d7aa-4969-9c60-e103cef65665-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.526061 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.526082 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.526480 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.526250 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.527000 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.526543 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.527000 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.526699 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.527000 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.526938 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.527270 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.527248 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.527331 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.527294 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cb51f3-9162-4372-b68e-5092fc67600e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.527390 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.527349 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0058a65d-d7aa-4969-9c60-e103cef65665-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.528214 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.528186 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.528538 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.528501 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.528840 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.528823 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0058a65d-d7aa-4969-9c60-e103cef65665-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.529739 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.529723 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cb51f3-9162-4372-b68e-5092fc67600e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.585014 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.584980 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.585014 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.585004 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76z5\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-kube-api-access-j76z5\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.585228 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.584980 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-prwvm\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.585228 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.584980 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tdq\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-kube-api-access-p2tdq\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n8gdm\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:55.617661 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.617635 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:55.873019 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:55.872939 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:56.035327 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.035302 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm"] Nov 01 03:58:56.036778 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:58:56.036751 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0058a65d_d7aa_4969_9c60_e103cef65665.slice/crio-53f790ff3609da2c8f8a1a0b2a8955cdd1dba330bed105a86a5c59ef01a62323 WatchSource:0}: Error finding container 53f790ff3609da2c8f8a1a0b2a8955cdd1dba330bed105a86a5c59ef01a62323: Status 404 returned error can't find the container with id 53f790ff3609da2c8f8a1a0b2a8955cdd1dba330bed105a86a5c59ef01a62323 Nov 01 03:58:56.039132 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.039100 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:56.039213 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.039167 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:56.039213 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.039197 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:56.092634 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.092605 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" event={"ID":"0058a65d-d7aa-4969-9c60-e103cef65665","Type":"ContainerStarted","Data":"53f790ff3609da2c8f8a1a0b2a8955cdd1dba330bed105a86a5c59ef01a62323"} Nov 01 03:58:56.242687 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.242663 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm"] Nov 01 03:58:56.244210 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:58:56.244185 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cb51f3_9162_4372_b68e_5092fc67600e.slice/crio-3c3b235b1c4921ff1f03dfb8ab5ea80ec1b794bc1c4651d7a664d3f8f9f82d21 WatchSource:0}: Error finding container 3c3b235b1c4921ff1f03dfb8ab5ea80ec1b794bc1c4651d7a664d3f8f9f82d21: Status 404 returned error can't find the container with id 3c3b235b1c4921ff1f03dfb8ab5ea80ec1b794bc1c4651d7a664d3f8f9f82d21 Nov 01 03:58:56.246590 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.246551 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:56.246698 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.246621 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:56.246698 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:56.246648 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 03:58:57.102289 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.102259 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" event={"ID":"0058a65d-d7aa-4969-9c60-e103cef65665","Type":"ContainerStarted","Data":"421da949b99f809efc3e8813c14c2984e7d8901682d41f0d690af8f42bf92a4b"} Nov 01 03:58:57.103884 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.103858 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" event={"ID":"22cb51f3-9162-4372-b68e-5092fc67600e","Type":"ContainerStarted","Data":"9294b869b56f12349ba67a46c1630e7eedbd66578300f40d13dd244b65e2dc8f"} Nov 01 03:58:57.104012 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.103897 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" event={"ID":"22cb51f3-9162-4372-b68e-5092fc67600e","Type":"ContainerStarted","Data":"3c3b235b1c4921ff1f03dfb8ab5ea80ec1b794bc1c4651d7a664d3f8f9f82d21"} Nov 01 03:58:57.322235 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.322179 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podStartSLOduration=2.322164164 podStartE2EDuration="2.322164164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:57.275532363 +0000 UTC m=+948.864627003" watchObservedRunningTime="2026-11-01 03:58:57.322164164 +0000 UTC m=+948.911258793" Nov 01 03:58:57.392058 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.388986 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podStartSLOduration=2.388967678 podStartE2EDuration="2.388967678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:57.388530536 +0000 UTC m=+948.977625166" watchObservedRunningTime="2026-11-01 03:58:57.388967678 +0000 UTC m=+948.978062309" Nov 01 03:58:57.618673 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.618634 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:58:57.619869 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.619848 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 03:58:57.619977 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.619892 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 03:58:57.877073 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.875557 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:58:57.881077 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.880005 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 03:58:57.881077 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:57.880087 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 03:58:58.618173 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:58.618134 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 03:58:58.618634 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:58.618610 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 03:58:58.889123 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:58.885275 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 03:58:58.889123 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:58.885335 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 03:58:59.618537 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:59.618498 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 03:58:59.618927 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:59.618560 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 03:58:59.873588 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:59.873490 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 03:58:59.873848 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:58:59.873821 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 03:59:00.618771 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:00.618732 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 03:59:00.619229 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:00.618793 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 03:59:00.874570 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:00.874476 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 03:59:00.874835 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:00.874804 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 03:59:01.617977 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:01.617943 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 03:59:01.618205 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:01.618001 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 03:59:01.877113 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:01.876415 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 03:59:01.877113 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:01.876474 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 03:59:02.009509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:02.009474 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 03:59:02.618873 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:02.618834 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 03:59:02.619114 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:02.618895 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 03:59:02.874020 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:02.873930 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 03:59:02.874020 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:02.873981 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 03:59:03.002053 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:03.001996 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:59:03.618078 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:03.618020 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 01 03:59:03.618295 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:03.618101 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 01 03:59:03.874104 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:03.873995 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 03:59:03.874263 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:03.874100 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 03:59:04.624265 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:04.624235 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:59:04.624779 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:04.624586 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:59:04.625407 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:04.625386 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 03:59:04.878171 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:04.878102 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:59:04.878407 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:04.878388 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:59:04.878994 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:04.878977 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 03:59:12.010560 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:12.010518 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 03:59:17.229341 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:17.229302 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:59:20.641658 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.641621 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9"] Nov 01 03:59:20.645434 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.645412 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.698750 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.698697 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"kube-root-ca.crt\"" Nov 01 03:59:20.698750 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.698718 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"openshift-service-ca.crt\"" Nov 01 03:59:20.741326 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.741290 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"default-dockercfg-fvxkt\"" Nov 01 03:59:20.741517 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.741331 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"distinct-w4-a9c5b3-kserve-self-signed-certs\"" Nov 01 03:59:20.743481 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.743452 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9"] Nov 01 03:59:20.745967 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.745950 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-dshm\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.746016 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.745990 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/639efb74-30dd-48a5-a350-f0c9210a16f6-kube-api-access-rpcdk\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.746131 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.746115 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-model-cache\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.746175 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.746158 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639efb74-30dd-48a5-a350-f0c9210a16f6-tls-certs\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.746290 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.746274 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-home\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.746328 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.746316 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-tmp-dir\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.847845 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.847813 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-model-cache\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848110 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.847864 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639efb74-30dd-48a5-a350-f0c9210a16f6-tls-certs\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848110 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.847929 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-home\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848110 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.848087 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-tmp-dir\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848319 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.848128 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-dshm\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848319 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.848155 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/639efb74-30dd-48a5-a350-f0c9210a16f6-kube-api-access-rpcdk\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848319 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.848279 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-model-cache\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848421 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.848317 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-home\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.848421 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.848357 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-tmp-dir\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.850440 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.850422 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-dshm\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.850559 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.850543 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639efb74-30dd-48a5-a350-f0c9210a16f6-tls-certs\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.906901 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.906825 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/639efb74-30dd-48a5-a350-f0c9210a16f6-kube-api-access-rpcdk\") pod \"distinct-w4-a9c5b3-kserve-6df8df868-vwkc9\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:20.957725 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:20.957692 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:21.385333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:21.385307 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9"] Nov 01 03:59:21.388904 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:59:21.388870 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod639efb74_30dd_48a5_a350_f0c9210a16f6.slice/crio-624b6b69543d459c53148dce4a75ca0910f79836c87dc394db0e44c28c48898c WatchSource:0}: Error finding container 624b6b69543d459c53148dce4a75ca0910f79836c87dc394db0e44c28c48898c: Status 404 returned error can't find the container with id 624b6b69543d459c53148dce4a75ca0910f79836c87dc394db0e44c28c48898c Nov 01 03:59:22.009774 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:22.009728 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 03:59:22.240531 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:22.240495 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" event={"ID":"639efb74-30dd-48a5-a350-f0c9210a16f6","Type":"ContainerStarted","Data":"73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5"} Nov 01 03:59:22.240736 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:22.240538 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" event={"ID":"639efb74-30dd-48a5-a350-f0c9210a16f6","Type":"ContainerStarted","Data":"624b6b69543d459c53148dce4a75ca0910f79836c87dc394db0e44c28c48898c"} Nov 01 03:59:22.351640 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:22.351535 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" podStartSLOduration=2.351519888 podStartE2EDuration="2.351519888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:59:22.350331006 +0000 UTC m=+973.939425659" watchObservedRunningTime="2026-11-01 03:59:22.351519888 +0000 UTC m=+973.940614516" Nov 01 03:59:22.958822 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:22.958780 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:30.958438 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:30.958406 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:30.970918 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:30.970892 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:31.302452 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:31.302372 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 03:59:32.010313 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:32.010266 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 03:59:33.000413 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:33.000374 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 03:59:35.336509 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.336474 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d44b46b89-qvj52"] Nov 01 03:59:35.337782 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.337750 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 03:59:35.372639 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.372609 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-kg28v"] Nov 01 03:59:35.377688 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.377672 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.443532 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.443490 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-kg28v"] Nov 01 03:59:35.509122 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.509091 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-plugins-config-volume\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.509312 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.509163 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-provider-credentials\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.509365 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.509318 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-routing-overlay-volume\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.509365 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.509357 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t65g\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-kube-api-access-7t65g\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.610023 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.609933 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-provider-credentials\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.610239 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.610077 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-routing-overlay-volume\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.610239 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.610110 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t65g\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-kube-api-access-7t65g\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.610239 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.610152 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-plugins-config-volume\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.610902 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.610877 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-routing-overlay-volume\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.611273 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.611248 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-plugins-config-volume\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.612649 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.612630 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-provider-credentials\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.668412 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.668382 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t65g\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-kube-api-access-7t65g\") pod \"payload-processing-external-model-6698d5675d-kg28v\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:35.692104 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:35.692068 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:36.064641 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.064614 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-kg28v"] Nov 01 03:59:36.067143 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:59:36.067109 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12431623_52e9_4ced_8f40_f0d8a0db79dd.slice/crio-7c0b30db5d045a7e4628f6a9278762b305fc99dd01511b03e971f83ebedc4a53 WatchSource:0}: Error finding container 7c0b30db5d045a7e4628f6a9278762b305fc99dd01511b03e971f83ebedc4a53: Status 404 returned error can't find the container with id 7c0b30db5d045a7e4628f6a9278762b305fc99dd01511b03e971f83ebedc4a53 Nov 01 03:59:36.075047 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.075013 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v"] Nov 01 03:59:36.080325 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.080307 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.129680 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.129627 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"default-dockercfg-qwscg\"" Nov 01 03:59:36.129680 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.129626 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"kube-root-ca.crt\"" Nov 01 03:59:36.173089 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.173058 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v"] Nov 01 03:59:36.237072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.227789 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-home\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.237072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.227839 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f64b8-e0f8-485d-9e23-2634c141d573-tls-certs\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.237072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.227914 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/ba0f64b8-e0f8-485d-9e23-2634c141d573-kube-api-access-vnhlx\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.237072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.227942 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-tmp-dir\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.237072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.228119 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-model-cache\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.237072 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.228153 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-dshm\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.241068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.239311 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"e2e-conflict-b-9e9797da-kserve-self-signed-certs\"" Nov 01 03:59:36.241068 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.239673 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"openshift-service-ca.crt\"" Nov 01 03:59:36.329106 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.329002 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-model-cache\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.329106 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.329089 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-dshm\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.329333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.329155 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-home\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.329333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.329181 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f64b8-e0f8-485d-9e23-2634c141d573-tls-certs\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.329333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.329259 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/ba0f64b8-e0f8-485d-9e23-2634c141d573-kube-api-access-vnhlx\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.329333 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.329285 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-tmp-dir\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.331755 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.330719 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-model-cache\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.331755 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.331025 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-home\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.331755 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.331116 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-tmp-dir\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.332317 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.332287 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-dshm\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.333406 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.333380 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f64b8-e0f8-485d-9e23-2634c141d573-tls-certs\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.334202 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.334172 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" event={"ID":"12431623-52e9-4ced-8f40-f0d8a0db79dd","Type":"ContainerStarted","Data":"14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09"} Nov 01 03:59:36.334305 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.334229 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" event={"ID":"12431623-52e9-4ced-8f40-f0d8a0db79dd","Type":"ContainerStarted","Data":"7c0b30db5d045a7e4628f6a9278762b305fc99dd01511b03e971f83ebedc4a53"} Nov 01 03:59:36.334363 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.334350 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 03:59:36.387340 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.387308 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/ba0f64b8-e0f8-485d-9e23-2634c141d573-kube-api-access-vnhlx\") pod \"e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.406168 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.406142 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:36.458717 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.456524 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" podStartSLOduration=1.4565008640000001 podStartE2EDuration="1.456500864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:59:36.446455555 +0000 UTC m=+988.035550184" watchObservedRunningTime="2026-11-01 03:59:36.456500864 +0000 UTC m=+988.045595492" Nov 01 03:59:36.825433 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:36.825406 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v"] Nov 01 03:59:36.827639 ip-10-0-128-69 kubenswrapper[6735]: W1101 03:59:36.827610 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba0f64b8_e0f8_485d_9e23_2634c141d573.slice/crio-186f34b6f55aae59cfcea025bf7c8d54349b5fdb3fb70d3e943b937aced1c75b WatchSource:0}: Error finding container 186f34b6f55aae59cfcea025bf7c8d54349b5fdb3fb70d3e943b937aced1c75b: Status 404 returned error can't find the container with id 186f34b6f55aae59cfcea025bf7c8d54349b5fdb3fb70d3e943b937aced1c75b Nov 01 03:59:37.340672 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:37.340633 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" event={"ID":"ba0f64b8-e0f8-485d-9e23-2634c141d573","Type":"ContainerStarted","Data":"f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06"} Nov 01 03:59:37.340867 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:37.340681 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" event={"ID":"ba0f64b8-e0f8-485d-9e23-2634c141d573","Type":"ContainerStarted","Data":"186f34b6f55aae59cfcea025bf7c8d54349b5fdb3fb70d3e943b937aced1c75b"} Nov 01 03:59:37.406976 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:37.406947 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:37.451892 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:37.451835 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" podStartSLOduration=1.4518200430000001 podStartE2EDuration="1.451820043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:59:37.450441233 +0000 UTC m=+989.039535863" watchObservedRunningTime="2026-11-01 03:59:37.451820043 +0000 UTC m=+989.040914672" Nov 01 03:59:40.337657 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.337599 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" containerID="cri-o://cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5" gracePeriod=25 Nov 01 03:59:40.587556 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.587529 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:59:40.684696 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.684669 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-provider-credentials\") pod \"aa2808c7-2ff6-4571-893b-9b705fe01423\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " Nov 01 03:59:40.684881 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.684787 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-routing-overlay-volume\") pod \"aa2808c7-2ff6-4571-893b-9b705fe01423\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " Nov 01 03:59:40.684881 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.684830 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-kube-api-access-8k464\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-kube-api-access-8k464\") pod \"aa2808c7-2ff6-4571-893b-9b705fe01423\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " Nov 01 03:59:40.684881 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.684870 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-plugins-config-volume\") pod \"aa2808c7-2ff6-4571-893b-9b705fe01423\" (UID: \"aa2808c7-2ff6-4571-893b-9b705fe01423\") " Nov 01 03:59:40.685259 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.685223 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-routing-overlay-volume" pod "aa2808c7-2ff6-4571-893b-9b705fe01423" (UID: "aa2808c7-2ff6-4571-893b-9b705fe01423"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:59:40.685396 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.685311 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-plugins-config-volume" pod "aa2808c7-2ff6-4571-893b-9b705fe01423" (UID: "aa2808c7-2ff6-4571-893b-9b705fe01423"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 03:59:40.687200 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.687179 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-provider-credentials" pod "aa2808c7-2ff6-4571-893b-9b705fe01423" (UID: "aa2808c7-2ff6-4571-893b-9b705fe01423"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:59:40.687285 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.687259 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-kube-api-access-8k464" pod "aa2808c7-2ff6-4571-893b-9b705fe01423" (UID: "aa2808c7-2ff6-4571-893b-9b705fe01423"). InnerVolumeSpecName "kube-api-access-8k464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 03:59:40.786648 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.786615 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:59:40.786648 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.786642 6735 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-provider-credentials\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:59:40.786648 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.786654 6735 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa2808c7-2ff6-4571-893b-9b705fe01423-routing-overlay-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:59:40.786880 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:40.786665 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k464\" (UniqueName: \"kubernetes.io/projected/aa2808c7-2ff6-4571-893b-9b705fe01423-kube-api-access-8k464\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 03:59:41.362722 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.362690 6735 generic.go:356] "Generic (PLEG): container finished" podID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerID="cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5" exitCode=0 Nov 01 03:59:41.363161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.362737 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" Nov 01 03:59:41.363161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.362778 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" event={"ID":"aa2808c7-2ff6-4571-893b-9b705fe01423","Type":"ContainerDied","Data":"cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5"} Nov 01 03:59:41.363161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.362830 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d44b46b89-qvj52" event={"ID":"aa2808c7-2ff6-4571-893b-9b705fe01423","Type":"ContainerDied","Data":"d34e45c2fb8f8b658ff030d9d73a815e8ce9138c6b4cf4bef0875cb780ff549c"} Nov 01 03:59:41.363161 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.362851 6735 scope.go:122] "RemoveContainer" containerID="cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5" Nov 01 03:59:41.371906 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.371889 6735 scope.go:122] "RemoveContainer" containerID="cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5" Nov 01 03:59:41.372176 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:41.372158 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5\": container with ID starting with cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5 not found: ID does not exist" containerID="cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5" Nov 01 03:59:41.372246 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.372186 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5"} err="failed to get container status \"cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5\": rpc error: code = NotFound desc = could not find container \"cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5\": container with ID starting with cefe10dc95f01bc45ef6d5471f92124f60de9b0f442f2e0d8b3115f6635dd7a5 not found: ID does not exist" Nov 01 03:59:41.517479 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.517446 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d44b46b89-qvj52"] Nov 01 03:59:41.563529 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:41.563499 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d44b46b89-qvj52"] Nov 01 03:59:42.538850 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:42.538811 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" path="/var/lib/kubelet/pods/aa2808c7-2ff6-4571-893b-9b705fe01423/volumes" Nov 01 03:59:42.906850 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:42.906762 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 03:59:46.408921 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:46.408886 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:46.505294 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:46.505266 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:46.640214 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:46.640178 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 03:59:47.344384 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:47.344346 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 03:59:48.636228 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:48.636189 6735 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1fec6c1a0815: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:48.636228 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:48.636220 6735 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:48.638809 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:48.636291 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials podName:34135bd6-9ca0-40ab-8745-f614b2644d39 nodeName:}" failed. No retries permitted until 2026-11-01 03:59:49.13627574 +0000 UTC m=+1000.725370347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials") pod "payload-processing-external-model-77d5dcc848-45zm6" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39") : secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:49.141829 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:49.141786 6735 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1fec6c1a0815: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:49.141829 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:49.141828 6735 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:49.142071 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:49.141883 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials podName:34135bd6-9ca0-40ab-8745-f614b2644d39 nodeName:}" failed. No retries permitted until 2026-11-01 03:59:50.141868175 +0000 UTC m=+1001.730962794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials") pod "payload-processing-external-model-77d5dcc848-45zm6" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39") : secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:50.153342 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:50.153306 6735 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1fec6c1a0815: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:50.153342 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:50.153340 6735 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:50.153770 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:50.153391 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials podName:34135bd6-9ca0-40ab-8745-f614b2644d39 nodeName:}" failed. No retries permitted until 2026-11-01 03:59:52.153377492 +0000 UTC m=+1003.742472099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials") pod "payload-processing-external-model-77d5dcc848-45zm6" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39") : secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:52.184774 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:52.184737 6735 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1fec6c1a0815: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:52.184774 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:52.184772 6735 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:52.185239 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:52.184834 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials podName:34135bd6-9ca0-40ab-8745-f614b2644d39 nodeName:}" failed. No retries permitted until 2026-11-01 03:59:56.184817754 +0000 UTC m=+1007.773912377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials") pod "payload-processing-external-model-77d5dcc848-45zm6" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39") : secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:56.231356 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:56.231313 6735 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1fec6c1a0815: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:56.231356 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:56.231357 6735 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6: secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:56.231770 ip-10-0-128-69 kubenswrapper[6735]: E1101 03:59:56.231412 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials podName:34135bd6-9ca0-40ab-8745-f614b2644d39 nodeName:}" failed. No retries permitted until 2026-11-01 04:00:04.231396719 +0000 UTC m=+1015.820491342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials") pod "payload-processing-external-model-77d5dcc848-45zm6" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39") : secret "agc-cred-1fec6c1a0815" not found Nov 01 03:59:57.342241 ip-10-0-128-69 kubenswrapper[6735]: I1101 03:59:57.342205 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:00:00.183268 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.183226 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p"] Nov 01 04:00:00.183806 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.183791 6735 cpu_manager.go:414] "containerMap: removing container" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" Nov 01 04:00:00.183849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.183807 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" Nov 01 04:00:00.183899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.183890 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2808c7-2ff6-4571-893b-9b705fe01423" containerName="payload-processing" Nov 01 04:00:00.187324 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.187308 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.236350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.236322 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\"" Nov 01 04:00:00.285591 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.285553 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p"] Nov 01 04:00:00.306368 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.306337 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj9k8\" (UniqueName: \"kubernetes.io/projected/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-kube-api-access-dj9k8\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.306561 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.306395 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.407401 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.407370 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9k8\" (UniqueName: \"kubernetes.io/projected/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-kube-api-access-dj9k8\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.407580 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.407428 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.408070 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.408031 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.465906 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.465879 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9k8\" (UniqueName: \"kubernetes.io/projected/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-kube-api-access-dj9k8\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.500998 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.500972 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:00.537203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.537157 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-kg28v"] Nov 01 04:00:00.538618 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.538586 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:00:00.554707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.554669 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8"] Nov 01 04:00:00.559090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.559069 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.608165 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.608138 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a763913535fa\"" Nov 01 04:00:00.608332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.608319 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-74b3b97eb424\"" Nov 01 04:00:00.615732 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.615689 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8"] Nov 01 04:00:00.711420 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.711384 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-provider-credentials\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.711420 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.711423 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvd7\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-kube-api-access-wlvd7\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.711697 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.711674 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-plugins-config-volume\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.711766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.711745 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-routing-overlay-volume\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.813078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.812965 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-routing-overlay-volume\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.813078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.813010 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-provider-credentials\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.813078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.813029 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvd7\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-kube-api-access-wlvd7\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.813374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.813249 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-plugins-config-volume\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.813685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.813662 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-routing-overlay-volume\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.813942 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.813925 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-plugins-config-volume\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.815752 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.815731 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-provider-credentials\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.871216 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.871190 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvd7\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-kube-api-access-wlvd7\") pod \"payload-processing-external-model-757c658c7b-pxlp8\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:00.877216 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.877187 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p"] Nov 01 04:00:00.879962 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:00.879937 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8422a04c_5bc5_4d94_ba8f_feab0f5221cd.slice/crio-3d49279525edf0a49fd7903b34a2d6bac4d1a1f34b67f2cf284bf0a067c3f718 WatchSource:0}: Error finding container 3d49279525edf0a49fd7903b34a2d6bac4d1a1f34b67f2cf284bf0a067c3f718: Status 404 returned error can't find the container with id 3d49279525edf0a49fd7903b34a2d6bac4d1a1f34b67f2cf284bf0a067c3f718 Nov 01 04:00:00.882099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.882081 6735 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 04:00:00.884002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:00.883981 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:01.253909 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:01.253878 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8"] Nov 01 04:00:01.255704 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:01.255680 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9ee1f3_bd87_43fe_8f45_c35ecac742f7.slice/crio-aa7a9b8f2a8d7a4a0d103c6535b951287dff988143c5bd0e877d3870fbfc0a44 WatchSource:0}: Error finding container aa7a9b8f2a8d7a4a0d103c6535b951287dff988143c5bd0e877d3870fbfc0a44: Status 404 returned error can't find the container with id aa7a9b8f2a8d7a4a0d103c6535b951287dff988143c5bd0e877d3870fbfc0a44 Nov 01 04:00:01.513801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:01.513705 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" event={"ID":"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7","Type":"ContainerStarted","Data":"77132ff0388ba70e7c5a13ca2c2ac1dd3d93856bce03388ec38ea9f50d80e491"} Nov 01 04:00:01.513801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:01.513774 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" event={"ID":"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7","Type":"ContainerStarted","Data":"aa7a9b8f2a8d7a4a0d103c6535b951287dff988143c5bd0e877d3870fbfc0a44"} Nov 01 04:00:01.514009 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:01.513833 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:01.515078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:01.515054 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" event={"ID":"8422a04c-5bc5-4d94-ba8f-feab0f5221cd","Type":"ContainerStarted","Data":"3d49279525edf0a49fd7903b34a2d6bac4d1a1f34b67f2cf284bf0a067c3f718"} Nov 01 04:00:01.625778 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:01.625722 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" podStartSLOduration=1.62570827 podStartE2EDuration="1.62570827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:01.623194544 +0000 UTC m=+1013.212289212" watchObservedRunningTime="2026-11-01 04:00:01.62570827 +0000 UTC m=+1013.214802899" Nov 01 04:00:03.001054 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:03.001007 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:00:04.270061 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:04.269234 6735 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1fec6c1a0815: secret "agc-cred-1fec6c1a0815" not found Nov 01 04:00:04.270061 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:04.269277 6735 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6: secret "agc-cred-1fec6c1a0815" not found Nov 01 04:00:04.270061 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:04.269342 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials podName:34135bd6-9ca0-40ab-8745-f614b2644d39 nodeName:}" failed. No retries permitted until 2026-11-01 04:00:20.269322407 +0000 UTC m=+1031.858417029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials") pod "payload-processing-external-model-77d5dcc848-45zm6" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39") : secret "agc-cred-1fec6c1a0815" not found Nov 01 04:00:05.539062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:05.536423 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:00:05.539062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:05.536490 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:00:05.539919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:05.539716 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerName="payload-processing" containerID="cri-o://14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09" gracePeriod=25 Nov 01 04:00:05.683789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:05.683738 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:00:05.684112 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:05.684076 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:00:05.928115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:05.927430 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:00:05.928115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:05.927495 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:00:06.242224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.242199 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 04:00:06.350666 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.350636 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-kube-api-access-7t65g\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-kube-api-access-7t65g\") pod \"12431623-52e9-4ced-8f40-f0d8a0db79dd\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " Nov 01 04:00:06.350852 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.350695 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-routing-overlay-volume\") pod \"12431623-52e9-4ced-8f40-f0d8a0db79dd\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " Nov 01 04:00:06.350852 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.350831 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-plugins-config-volume\") pod \"12431623-52e9-4ced-8f40-f0d8a0db79dd\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " Nov 01 04:00:06.350963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.350885 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-provider-credentials\") pod \"12431623-52e9-4ced-8f40-f0d8a0db79dd\" (UID: \"12431623-52e9-4ced-8f40-f0d8a0db79dd\") " Nov 01 04:00:06.351366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.351333 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-plugins-config-volume" pod "12431623-52e9-4ced-8f40-f0d8a0db79dd" (UID: "12431623-52e9-4ced-8f40-f0d8a0db79dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:00:06.351534 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.351499 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-routing-overlay-volume" pod "12431623-52e9-4ced-8f40-f0d8a0db79dd" (UID: "12431623-52e9-4ced-8f40-f0d8a0db79dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:00:06.354090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.354065 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-kube-api-access-7t65g" pod "12431623-52e9-4ced-8f40-f0d8a0db79dd" (UID: "12431623-52e9-4ced-8f40-f0d8a0db79dd"). InnerVolumeSpecName "kube-api-access-7t65g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:00:06.354090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.354067 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-provider-credentials" pod "12431623-52e9-4ced-8f40-f0d8a0db79dd" (UID: "12431623-52e9-4ced-8f40-f0d8a0db79dd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:00:06.452563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.452525 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:06.452563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.452556 6735 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-provider-credentials\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:06.452563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.452566 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t65g\" (UniqueName: \"kubernetes.io/projected/12431623-52e9-4ced-8f40-f0d8a0db79dd-kube-api-access-7t65g\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:06.452563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.452574 6735 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12431623-52e9-4ced-8f40-f0d8a0db79dd-routing-overlay-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:06.620803 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.620770 6735 generic.go:356] "Generic (PLEG): container finished" podID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerID="14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09" exitCode=0 Nov 01 04:00:06.621319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.620819 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" Nov 01 04:00:06.621319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.620860 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" event={"ID":"12431623-52e9-4ced-8f40-f0d8a0db79dd","Type":"ContainerDied","Data":"14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09"} Nov 01 04:00:06.621319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.620915 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-kg28v" event={"ID":"12431623-52e9-4ced-8f40-f0d8a0db79dd","Type":"ContainerDied","Data":"7c0b30db5d045a7e4628f6a9278762b305fc99dd01511b03e971f83ebedc4a53"} Nov 01 04:00:06.621319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.620947 6735 scope.go:122] "RemoveContainer" containerID="14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09" Nov 01 04:00:06.630849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.630827 6735 scope.go:122] "RemoveContainer" containerID="14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09" Nov 01 04:00:06.631171 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:06.631149 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09\": container with ID starting with 14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09 not found: ID does not exist" containerID="14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09" Nov 01 04:00:06.631235 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.631181 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09"} err="failed to get container status \"14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09\": rpc error: code = NotFound desc = could not find container \"14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09\": container with ID starting with 14aaae2d546c36aa787948873eec4d171054ec502efd8bfb543a9a4a3c472a09 not found: ID does not exist" Nov 01 04:00:06.775383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.775349 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-kg28v"] Nov 01 04:00:06.822112 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:06.822002 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-kg28v"] Nov 01 04:00:08.525932 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:08.525901 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" path="/var/lib/kubelet/pods/12431623-52e9-4ced-8f40-f0d8a0db79dd/volumes" Nov 01 04:00:09.649703 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:09.649613 6735 generic.go:356] "Generic (PLEG): container finished" podID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerID="e43fb68a221fc989aa461dcc005c0c7310d21da5524ae2ec129cf4a9d5fb4700" exitCode=60 Nov 01 04:00:09.649703 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:09.649662 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" event={"ID":"8422a04c-5bc5-4d94-ba8f-feab0f5221cd","Type":"ContainerDied","Data":"e43fb68a221fc989aa461dcc005c0c7310d21da5524ae2ec129cf4a9d5fb4700"} Nov 01 04:00:09.650128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:09.649951 6735 scope.go:122] "RemoveContainer" containerID="e43fb68a221fc989aa461dcc005c0c7310d21da5524ae2ec129cf4a9d5fb4700" Nov 01 04:00:10.656821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:10.656784 6735 generic.go:356] "Generic (PLEG): container finished" podID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerID="697ec8eef78d5ba961a41e0c1f837da9a3896758f60eb75bcd7b0029e794bb3f" exitCode=60 Nov 01 04:00:10.657233 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:10.656865 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" event={"ID":"8422a04c-5bc5-4d94-ba8f-feab0f5221cd","Type":"ContainerDied","Data":"697ec8eef78d5ba961a41e0c1f837da9a3896758f60eb75bcd7b0029e794bb3f"} Nov 01 04:00:10.657233 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:10.656923 6735 scope.go:122] "RemoveContainer" containerID="e43fb68a221fc989aa461dcc005c0c7310d21da5524ae2ec129cf4a9d5fb4700" Nov 01 04:00:10.657233 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:10.657221 6735 scope.go:122] "RemoveContainer" containerID="697ec8eef78d5ba961a41e0c1f837da9a3896758f60eb75bcd7b0029e794bb3f" Nov 01 04:00:10.657515 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:10.657491 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p_odh-ai-gateway-infra(8422a04c-5bc5-4d94-ba8f-feab0f5221cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" Nov 01 04:00:11.665902 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:11.665873 6735 scope.go:122] "RemoveContainer" containerID="697ec8eef78d5ba961a41e0c1f837da9a3896758f60eb75bcd7b0029e794bb3f" Nov 01 04:00:11.666482 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:11.666167 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p_odh-ai-gateway-infra(8422a04c-5bc5-4d94-ba8f-feab0f5221cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" Nov 01 04:00:12.525679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:12.525650 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:00:12.684502 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:12.684471 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6"] Nov 01 04:00:17.686022 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:17.685962 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" podUID="34135bd6-9ca0-40ab-8745-f614b2644d39" containerName="payload-processing" containerID="cri-o://ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f" gracePeriod=25 Nov 01 04:00:17.865319 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:17.865293 6735 logging.go:55] [core] [Channel #1316 SubChannel #1317]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.58:50052", ServerName: "10.134.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.58:50052: connect: connection refused" Nov 01 04:00:17.984224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:17.984198 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 04:00:18.077556 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.077523 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-routing-overlay-volume\") pod \"34135bd6-9ca0-40ab-8745-f614b2644d39\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " Nov 01 04:00:18.077747 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.077630 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials\") pod \"34135bd6-9ca0-40ab-8745-f614b2644d39\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " Nov 01 04:00:18.077747 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.077673 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-plugins-config-volume\") pod \"34135bd6-9ca0-40ab-8745-f614b2644d39\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " Nov 01 04:00:18.077869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.077748 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-kube-api-access-zhhc4\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-kube-api-access-zhhc4\") pod \"34135bd6-9ca0-40ab-8745-f614b2644d39\" (UID: \"34135bd6-9ca0-40ab-8745-f614b2644d39\") " Nov 01 04:00:18.077977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.077950 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-routing-overlay-volume" pod "34135bd6-9ca0-40ab-8745-f614b2644d39" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:00:18.078088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.078068 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-plugins-config-volume" pod "34135bd6-9ca0-40ab-8745-f614b2644d39" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:00:18.078222 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.078210 6735 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-routing-overlay-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:18.078260 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.078224 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34135bd6-9ca0-40ab-8745-f614b2644d39-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:18.079972 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.079948 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-kube-api-access-zhhc4" pod "34135bd6-9ca0-40ab-8745-f614b2644d39" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39"). InnerVolumeSpecName "kube-api-access-zhhc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:00:18.080069 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.079988 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials" pod "34135bd6-9ca0-40ab-8745-f614b2644d39" (UID: "34135bd6-9ca0-40ab-8745-f614b2644d39"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:00:18.178891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.178858 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhhc4\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-kube-api-access-zhhc4\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:18.178891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.178887 6735 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34135bd6-9ca0-40ab-8745-f614b2644d39-provider-credentials\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:18.579714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.579667 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn"] Nov 01 04:00:18.581431 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.581394 6735 cpu_manager.go:414] "containerMap: removing container" podUID="34135bd6-9ca0-40ab-8745-f614b2644d39" containerName="payload-processing" Nov 01 04:00:18.581431 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.581419 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34135bd6-9ca0-40ab-8745-f614b2644d39" containerName="payload-processing" Nov 01 04:00:18.581431 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.581432 6735 cpu_manager.go:414] "containerMap: removing container" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerName="payload-processing" Nov 01 04:00:18.581919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.581439 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerName="payload-processing" Nov 01 04:00:18.581919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.581624 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="34135bd6-9ca0-40ab-8745-f614b2644d39" containerName="payload-processing" Nov 01 04:00:18.581919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.581642 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="12431623-52e9-4ced-8f40-f0d8a0db79dd" containerName="payload-processing" Nov 01 04:00:18.585385 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.585365 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.647966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.647930 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx"] Nov 01 04:00:18.652109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.652083 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.658322 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.658298 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn"] Nov 01 04:00:18.684428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684397 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.684687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684658 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.684813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684722 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.684813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684788 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.684963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684815 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.684963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684851 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.684963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684878 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87jn\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-kube-api-access-b87jn\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.684963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684922 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.685209 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.684972 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.689317 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.688993 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx"] Nov 01 04:00:18.710607 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.710574 6735 generic.go:356] "Generic (PLEG): container finished" podID="34135bd6-9ca0-40ab-8745-f614b2644d39" containerID="ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f" exitCode=0 Nov 01 04:00:18.710764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.710653 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" Nov 01 04:00:18.710764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.710662 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" event={"ID":"34135bd6-9ca0-40ab-8745-f614b2644d39","Type":"ContainerDied","Data":"ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f"} Nov 01 04:00:18.710764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.710735 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" event={"ID":"34135bd6-9ca0-40ab-8745-f614b2644d39","Type":"ContainerDied","Data":"d7aa8ce06fb9dc3f63af2eddb125ddcf9b5925ce1167186c5bff624fda20253e"} Nov 01 04:00:18.711023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.710765 6735 scope.go:122] "RemoveContainer" containerID="ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f" Nov 01 04:00:18.720327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.720308 6735 scope.go:122] "RemoveContainer" containerID="ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f" Nov 01 04:00:18.720634 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:18.720616 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f\": container with ID starting with ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f not found: ID does not exist" containerID="ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f" Nov 01 04:00:18.720696 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.720640 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f"} err="failed to get container status \"ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f\": rpc error: code = NotFound desc = could not find container \"ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f\": container with ID starting with ac04645311f6523b8e9ae1ae4c95cdbeb27bfdd5b597c3f9d22834658daddb7f not found: ID does not exist" Nov 01 04:00:18.785754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.785723 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdqv\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-kube-api-access-zmdqv\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.785945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.785763 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.785945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.785791 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.785945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.785874 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.786118 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.785943 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786118 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786009 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786118 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786070 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.786265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786157 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786214 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786255 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786262 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3642c443-e753-4efe-871f-e55be1536776-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.786369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786331 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786336 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786359 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87jn\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-kube-api-access-b87jn\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786416 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786454 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786467 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.786650 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786521 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.786650 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786551 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.786650 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786593 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.786650 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786620 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.786804 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.786667 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3642c443-e753-4efe-871f-e55be1536776-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.787060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.787015 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.788505 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.788450 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.789084 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.789068 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.843827 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.843796 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.843995 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.843974 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87jn\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-kube-api-access-b87jn\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.865576 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.865545 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6" podUID="34135bd6-9ca0-40ab-8745-f614b2644d39" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.58:50052\" within 1s: context deadline exceeded" Nov 01 04:00:18.865705 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:18.865660 6735 logging.go:55] [core] [Channel #1316 SubChannel #1317]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.58:50052", ServerName: "10.134.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.58:50052: operation was canceled" Nov 01 04:00:18.887450 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887423 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887457 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887493 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887511 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887531 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3642c443-e753-4efe-871f-e55be1536776-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887591 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdqv\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-kube-api-access-zmdqv\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887657 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887726 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887751 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3642c443-e753-4efe-871f-e55be1536776-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887808 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-credential-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.887884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.887836 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-data\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.888208 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.888190 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-certs\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.888289 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.888172 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-socket\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.888624 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.888553 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3642c443-e753-4efe-871f-e55be1536776-istiod-ca-cert\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.889972 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.889952 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-envoy\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.890457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.890441 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3642c443-e753-4efe-871f-e55be1536776-istio-podinfo\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.899931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.899792 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:18.923808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.923783 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6"] Nov 01 04:00:18.929792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.929765 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-77d5dcc848-45zm6"] Nov 01 04:00:18.945627 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.945597 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-istio-token\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.945750 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.945661 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdqv\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-kube-api-access-zmdqv\") pod \"e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:18.968418 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:18.966216 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:19.319436 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.319391 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn"] Nov 01 04:00:19.353781 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:19.353752 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe46460e_0ced_4bd2_9a3c_b4f9df8385d3.slice/crio-257df16a8b517660b0bc049195eca8d6f0a08ab7abf83b7cec86f3c68d5ea68f WatchSource:0}: Error finding container 257df16a8b517660b0bc049195eca8d6f0a08ab7abf83b7cec86f3c68d5ea68f: Status 404 returned error can't find the container with id 257df16a8b517660b0bc049195eca8d6f0a08ab7abf83b7cec86f3c68d5ea68f Nov 01 04:00:19.355979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.355948 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:19.356070 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.356016 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:19.356115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.356067 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:19.370547 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.370514 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx"] Nov 01 04:00:19.372066 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:19.372026 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3642c443_e753_4efe_871f_e55be1536776.slice/crio-b700b7322bc1acd79210a48ff319fc1ae075655b184b4d1119bad402ceacfec6 WatchSource:0}: Error finding container b700b7322bc1acd79210a48ff319fc1ae075655b184b4d1119bad402ceacfec6: Status 404 returned error can't find the container with id b700b7322bc1acd79210a48ff319fc1ae075655b184b4d1119bad402ceacfec6 Nov 01 04:00:19.374593 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.374558 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:19.374681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.374623 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:19.374681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.374655 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:19.719876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.719837 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" event={"ID":"3642c443-e753-4efe-871f-e55be1536776","Type":"ContainerStarted","Data":"aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6"} Nov 01 04:00:19.720362 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.719907 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" event={"ID":"3642c443-e753-4efe-871f-e55be1536776","Type":"ContainerStarted","Data":"b700b7322bc1acd79210a48ff319fc1ae075655b184b4d1119bad402ceacfec6"} Nov 01 04:00:19.721754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.721730 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" event={"ID":"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3","Type":"ContainerStarted","Data":"2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12"} Nov 01 04:00:19.721875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.721774 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" event={"ID":"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3","Type":"ContainerStarted","Data":"257df16a8b517660b0bc049195eca8d6f0a08ab7abf83b7cec86f3c68d5ea68f"} Nov 01 04:00:19.833641 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.833575 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podStartSLOduration=1.833556947 podStartE2EDuration="1.833556947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:19.833140297 +0000 UTC m=+1031.422234921" watchObservedRunningTime="2026-11-01 04:00:19.833556947 +0000 UTC m=+1031.422651578" Nov 01 04:00:19.905060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.904146 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:19.938208 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.938161 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:00:19.975122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:19.973121 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:20.527985 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:20.527938 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34135bd6-9ca0-40ab-8745-f614b2644d39" path="/var/lib/kubelet/pods/34135bd6-9ca0-40ab-8745-f614b2644d39/volumes" Nov 01 04:00:20.981507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:20.979459 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:20.981507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:20.979521 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:21.013062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:20.995654 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:21.013062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:20.995716 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:22.121061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.120164 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:22.121061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.120229 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:22.121061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.120645 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:22.121061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.120681 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:22.520061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.519549 6735 scope.go:122] "RemoveContainer" containerID="697ec8eef78d5ba961a41e0c1f837da9a3896758f60eb75bcd7b0029e794bb3f" Nov 01 04:00:22.902113 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.901902 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:22.902113 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.902066 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:22.971088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.970465 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:22.971088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:22.970528 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:23.894083 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.889706 6735 generic.go:356] "Generic (PLEG): container finished" podID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerID="3c7d0c6848d1ef10505c45fbe8850d056265e959f8239241e1a74040bcd7e2ae" exitCode=60 Nov 01 04:00:23.894719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.894100 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" event={"ID":"8422a04c-5bc5-4d94-ba8f-feab0f5221cd","Type":"ContainerDied","Data":"3c7d0c6848d1ef10505c45fbe8850d056265e959f8239241e1a74040bcd7e2ae"} Nov 01 04:00:23.894906 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.894892 6735 scope.go:122] "RemoveContainer" containerID="697ec8eef78d5ba961a41e0c1f837da9a3896758f60eb75bcd7b0029e794bb3f" Nov 01 04:00:23.899585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.899563 6735 scope.go:122] "RemoveContainer" containerID="3c7d0c6848d1ef10505c45fbe8850d056265e959f8239241e1a74040bcd7e2ae" Nov 01 04:00:23.900103 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:23.900076 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p_odh-ai-gateway-infra(8422a04c-5bc5-4d94-ba8f-feab0f5221cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" Nov 01 04:00:23.906822 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.906798 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:23.906996 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.906974 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:23.967429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.967392 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:23.967701 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:23.967673 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:24.017999 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:24.017858 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podStartSLOduration=6.017838699 podStartE2EDuration="6.017838699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:19.971849432 +0000 UTC m=+1031.560944062" watchObservedRunningTime="2026-11-01 04:00:24.017838699 +0000 UTC m=+1035.606933330" Nov 01 04:00:24.914063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:24.913493 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:24.914063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:24.913564 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:24.968059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:24.967470 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:24.968059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:24.967540 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:25.053140 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:25.053100 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p"] Nov 01 04:00:25.917528 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:25.917489 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:25.917967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:25.917557 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:25.934623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:25.934598 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:25.996060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:25.979144 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:25.996060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:25.979222 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:26.004472 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.004309 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" Nov 01 04:00:26.004472 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.004309 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p" event={"ID":"8422a04c-5bc5-4d94-ba8f-feab0f5221cd","Type":"ContainerDied","Data":"3d49279525edf0a49fd7903b34a2d6bac4d1a1f34b67f2cf284bf0a067c3f718"} Nov 01 04:00:26.004472 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.004444 6735 scope.go:122] "RemoveContainer" containerID="3c7d0c6848d1ef10505c45fbe8850d056265e959f8239241e1a74040bcd7e2ae" Nov 01 04:00:26.017851 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.017831 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-kube-api-access-dj9k8\" (UniqueName: \"kubernetes.io/projected/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-kube-api-access-dj9k8\") pod \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " Nov 01 04:00:26.017947 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.017860 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-maas-api-service-ca\") pod \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\" (UID: \"8422a04c-5bc5-4d94-ba8f-feab0f5221cd\") " Nov 01 04:00:26.018200 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.018180 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-maas-api-service-ca" pod "8422a04c-5bc5-4d94-ba8f-feab0f5221cd" (UID: "8422a04c-5bc5-4d94-ba8f-feab0f5221cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:00:26.020141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.020124 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-kube-api-access-dj9k8" pod "8422a04c-5bc5-4d94-ba8f-feab0f5221cd" (UID: "8422a04c-5bc5-4d94-ba8f-feab0f5221cd"). InnerVolumeSpecName "kube-api-access-dj9k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:00:26.119611 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.119574 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj9k8\" (UniqueName: \"kubernetes.io/projected/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-kube-api-access-dj9k8\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:26.119611 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.119604 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8422a04c-5bc5-4d94-ba8f-feab0f5221cd-maas-api-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:00:26.430981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.430946 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p"] Nov 01 04:00:26.443190 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.443161 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891760-qkj9p"] Nov 01 04:00:26.528388 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.528352 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" path="/var/lib/kubelet/pods/8422a04c-5bc5-4d94-ba8f-feab0f5221cd/volumes" Nov 01 04:00:26.900595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.900559 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:26.900782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.900625 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:26.973198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.973155 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:26.973820 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:26.973790 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:27.900773 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:27.900739 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 01 04:00:27.900941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:27.900790 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 01 04:00:27.967271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:27.967236 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 01 04:00:27.967426 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:27.967290 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 01 04:00:28.908894 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:28.908860 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:28.910122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:28.910098 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:28.911180 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:28.911162 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:00:28.993841 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:28.993810 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:28.994462 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:28.994438 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:28.995888 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:28.995864 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:00:33.001262 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:00:33.001223 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:00:51.276725 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:51.276687 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:00:56.635461 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.635428 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc"] Nov 01 04:00:56.636189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636168 6735 cpu_manager.go:414] "containerMap: removing container" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.636189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636188 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.636379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636220 6735 cpu_manager.go:414] "containerMap: removing container" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.636379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636227 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.636379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636255 6735 cpu_manager.go:414] "containerMap: removing container" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.636379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636262 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.636567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636399 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.636567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.636415 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:56.640013 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.639992 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.704565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.704531 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24gj\" (UniqueName: \"kubernetes.io/projected/ba97abfe-c233-4de8-83d9-6c050986102d-kube-api-access-v24gj\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.704728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.704599 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-home\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.704801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.704776 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-dshm\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.704879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.704862 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-model-cache\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.704959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.704933 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-tmp-dir\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.705068 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.705023 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba97abfe-c233-4de8-83d9-6c050986102d-tls-certs\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.735326 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.735292 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a9c5b3\"/\"distinct-2-w4-a9c5b3-kserve-self-signed-certs\"" Nov 01 04:00:56.737468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.737447 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc"] Nov 01 04:00:56.807212 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.807182 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-dshm\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.808194 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.808165 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-model-cache\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.808720 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.808701 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-tmp-dir\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.808886 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.808869 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba97abfe-c233-4de8-83d9-6c050986102d-tls-certs\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.814086 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.809084 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24gj\" (UniqueName: \"kubernetes.io/projected/ba97abfe-c233-4de8-83d9-6c050986102d-kube-api-access-v24gj\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.814319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.814296 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-home\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.817237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.808642 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-model-cache\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.817402 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.815422 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-home\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.817509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.815738 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-tmp-dir\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.829381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.829349 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-dshm\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.829894 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.829842 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba97abfe-c233-4de8-83d9-6c050986102d-tls-certs\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.872551 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.872520 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24gj\" (UniqueName: \"kubernetes.io/projected/ba97abfe-c233-4de8-83d9-6c050986102d-kube-api-access-v24gj\") pod \"distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:56.952164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:56.952087 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:57.397981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:57.397957 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc"] Nov 01 04:00:57.400064 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:57.400012 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba97abfe_c233_4de8_83d9_6c050986102d.slice/crio-253a34f6d9258ad2a7f5d479a8b3e9234265a318d92a61c0c13b1bcce62626e8 WatchSource:0}: Error finding container 253a34f6d9258ad2a7f5d479a8b3e9234265a318d92a61c0c13b1bcce62626e8: Status 404 returned error can't find the container with id 253a34f6d9258ad2a7f5d479a8b3e9234265a318d92a61c0c13b1bcce62626e8 Nov 01 04:00:58.190401 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:58.190366 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" event={"ID":"ba97abfe-c233-4de8-83d9-6c050986102d","Type":"ContainerStarted","Data":"72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865"} Nov 01 04:00:58.190401 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:58.190409 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" event={"ID":"ba97abfe-c233-4de8-83d9-6c050986102d","Type":"ContainerStarted","Data":"253a34f6d9258ad2a7f5d479a8b3e9234265a318d92a61c0c13b1bcce62626e8"} Nov 01 04:00:58.302208 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:58.302160 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" podStartSLOduration=2.302146616 podStartE2EDuration="2.302146616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:58.300829592 +0000 UTC m=+1069.889924220" watchObservedRunningTime="2026-11-01 04:00:58.302146616 +0000 UTC m=+1069.891241245" Nov 01 04:00:58.953160 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:58.953126 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:00:59.213421 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.213389 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv"] Nov 01 04:00:59.213969 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.213956 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="8422a04c-5bc5-4d94-ba8f-feab0f5221cd" containerName="cleanup" Nov 01 04:00:59.217290 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.217269 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.317648 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.317617 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv"] Nov 01 04:00:59.364893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.364854 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.364893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.364896 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.365111 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.364923 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.365111 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.365069 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.365194 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.365143 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbm2\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-kube-api-access-jwbm2\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.365194 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.365174 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.365274 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.365239 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.365404 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.365387 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931116-382f-4f97-83a1-90befaf22f2e-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.365489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.365467 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931116-382f-4f97-83a1-90befaf22f2e-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.466996 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467301 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467147 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931116-382f-4f97-83a1-90befaf22f2e-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467301 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467195 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931116-382f-4f97-83a1-90befaf22f2e-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467407 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467322 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467407 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467364 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467407 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467390 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467436 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467500 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbm2\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-kube-api-access-jwbm2\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467524 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467769 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467857 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467835 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.467937 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467918 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.468187 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.467957 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931116-382f-4f97-83a1-90befaf22f2e-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.468187 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.468100 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.469700 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.469680 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.470029 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.470012 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931116-382f-4f97-83a1-90befaf22f2e-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.524968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.524942 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.525180 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.525157 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbm2\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-kube-api-access-jwbm2\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.536306 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.536249 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:00:59.957163 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.957139 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv"] Nov 01 04:00:59.958704 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:00:59.958672 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b931116_382f_4f97_83a1_90befaf22f2e.slice/crio-5e21e905fc610bb52b2bff00c28cbc036313ec79c1da0b67573804acff4a0cce WatchSource:0}: Error finding container 5e21e905fc610bb52b2bff00c28cbc036313ec79c1da0b67573804acff4a0cce: Status 404 returned error can't find the container with id 5e21e905fc610bb52b2bff00c28cbc036313ec79c1da0b67573804acff4a0cce Nov 01 04:00:59.961127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.961094 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:59.961219 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.961156 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:00:59.961219 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:00:59.961184 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:00.204949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:00.204914 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" event={"ID":"3b931116-382f-4f97-83a1-90befaf22f2e","Type":"ContainerStarted","Data":"7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0"} Nov 01 04:01:00.205230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:00.204970 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" event={"ID":"3b931116-382f-4f97-83a1-90befaf22f2e","Type":"ContainerStarted","Data":"5e21e905fc610bb52b2bff00c28cbc036313ec79c1da0b67573804acff4a0cce"} Nov 01 04:01:00.320099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:00.319942 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podStartSLOduration=1.31992467 podStartE2EDuration="1.31992467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:00.318897833 +0000 UTC m=+1071.907992462" watchObservedRunningTime="2026-11-01 04:01:00.31992467 +0000 UTC m=+1071.909019304" Nov 01 04:01:00.536381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:00.536352 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:01:00.627957 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:00.625833 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v"] Nov 01 04:01:01.538706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:01.538671 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 04:01:01.539147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:01.538733 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 04:01:02.537511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:02.537475 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 04:01:02.537684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:02.537524 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 04:01:03.001678 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:01:03.001640 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:01:03.537340 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:03.537307 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 04:01:03.537520 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:03.537359 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 04:01:04.537517 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:04.537482 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 04:01:04.537877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:04.537535 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 04:01:05.353070 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.352804 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb"] Nov 01 04:01:05.374513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.374483 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.411308 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.411282 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb"] Nov 01 04:01:05.471536 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.471508 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-routing-overlay-volume\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.471719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.471553 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgsjd\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-kube-api-access-cgsjd\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.471719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.471624 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-provider-credentials\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.471719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.471686 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-plugins-config-volume\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.536920 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.536883 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 04:01:05.537126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.536951 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 04:01:05.573787 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.573751 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-plugins-config-volume\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.574319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.573942 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-routing-overlay-volume\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.574319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.574005 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgsjd\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-kube-api-access-cgsjd\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.574319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.574057 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-provider-credentials\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.575114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.575082 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-plugins-config-volume\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.575408 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.575387 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-routing-overlay-volume\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.577046 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.577013 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-provider-credentials\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.633211 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.633120 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgsjd\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-kube-api-access-cgsjd\") pod \"payload-processing-external-model-6698d5675d-tkzwb\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:05.688414 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:05.688371 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:06.120460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.111082 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb"] Nov 01 04:01:06.242545 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.242441 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" event={"ID":"00e7b1ed-7d66-4baa-989c-4656597cd10c","Type":"ContainerStarted","Data":"c9f5c604bbdba4738f95571fd6d6f58419a7ac2b15654b5599964518cce90d51"} Nov 01 04:01:06.242545 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.242501 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" event={"ID":"00e7b1ed-7d66-4baa-989c-4656597cd10c","Type":"ContainerStarted","Data":"6bcc4fe5850c9d8f534d45e4954c94a39d48e45bd54605c9f53a19d5594d568a"} Nov 01 04:01:06.242545 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.242531 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:06.358757 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.358669 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" podStartSLOduration=1.358649784 podStartE2EDuration="1.358649784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:06.35442475 +0000 UTC m=+1077.943519383" watchObservedRunningTime="2026-11-01 04:01:06.358649784 +0000 UTC m=+1077.947744413" Nov 01 04:01:06.575670 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.575169 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 01 04:01:06.575670 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.575236 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 01 04:01:06.966451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:06.966421 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:01:07.028871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:07.028842 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:01:07.376225 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:07.376138 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:01:07.895381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:07.895332 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:01:07.896067 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:07.896014 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:01:07.896904 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:07.896884 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:01:15.664653 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:15.664566 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" podUID="ba0f64b8-e0f8-485d-9e23-2634c141d573" containerName="main" containerID="cri-o://f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06" gracePeriod=45 Nov 01 04:01:15.921678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:15.921624 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 04:01:16.062632 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.062592 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-home\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-home\") pod \"ba0f64b8-e0f8-485d-9e23-2634c141d573\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " Nov 01 04:01:16.062821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.062650 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-dshm\") pod \"ba0f64b8-e0f8-485d-9e23-2634c141d573\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " Nov 01 04:01:16.062821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.062695 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0f64b8-e0f8-485d-9e23-2634c141d573-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f64b8-e0f8-485d-9e23-2634c141d573-tls-certs\") pod \"ba0f64b8-e0f8-485d-9e23-2634c141d573\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " Nov 01 04:01:16.062821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.062779 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0f64b8-e0f8-485d-9e23-2634c141d573-kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/ba0f64b8-e0f8-485d-9e23-2634c141d573-kube-api-access-vnhlx\") pod \"ba0f64b8-e0f8-485d-9e23-2634c141d573\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " Nov 01 04:01:16.062821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.062804 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-tmp-dir\") pod \"ba0f64b8-e0f8-485d-9e23-2634c141d573\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " Nov 01 04:01:16.063023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.062984 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-model-cache\") pod \"ba0f64b8-e0f8-485d-9e23-2634c141d573\" (UID: \"ba0f64b8-e0f8-485d-9e23-2634c141d573\") " Nov 01 04:01:16.063644 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.063579 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-tmp-dir" pod "ba0f64b8-e0f8-485d-9e23-2634c141d573" (UID: "ba0f64b8-e0f8-485d-9e23-2634c141d573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:01:16.063644 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.063599 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-model-cache" pod "ba0f64b8-e0f8-485d-9e23-2634c141d573" (UID: "ba0f64b8-e0f8-485d-9e23-2634c141d573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:01:16.063879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.062925 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-home" pod "ba0f64b8-e0f8-485d-9e23-2634c141d573" (UID: "ba0f64b8-e0f8-485d-9e23-2634c141d573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:01:16.065406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.065380 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-dshm" pod "ba0f64b8-e0f8-485d-9e23-2634c141d573" (UID: "ba0f64b8-e0f8-485d-9e23-2634c141d573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:01:16.065697 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.065672 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0f64b8-e0f8-485d-9e23-2634c141d573-tls-certs" pod "ba0f64b8-e0f8-485d-9e23-2634c141d573" (UID: "ba0f64b8-e0f8-485d-9e23-2634c141d573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:01:16.065823 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.065801 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0f64b8-e0f8-485d-9e23-2634c141d573-kube-api-access-vnhlx" pod "ba0f64b8-e0f8-485d-9e23-2634c141d573" (UID: "ba0f64b8-e0f8-485d-9e23-2634c141d573"). InnerVolumeSpecName "kube-api-access-vnhlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:01:16.164360 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.164325 6735 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-model-cache\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:16.164360 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.164353 6735 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-home\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:16.164360 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.164365 6735 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-dshm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:16.164628 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.164373 6735 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f64b8-e0f8-485d-9e23-2634c141d573-tls-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:16.164628 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.164382 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/ba0f64b8-e0f8-485d-9e23-2634c141d573-kube-api-access-vnhlx\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:16.164628 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.164391 6735 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba0f64b8-e0f8-485d-9e23-2634c141d573-tmp-dir\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:16.310293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.310264 6735 generic.go:356] "Generic (PLEG): container finished" podID="ba0f64b8-e0f8-485d-9e23-2634c141d573" containerID="f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06" exitCode=0 Nov 01 04:01:16.310475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.310309 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" Nov 01 04:01:16.310475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.310338 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" event={"ID":"ba0f64b8-e0f8-485d-9e23-2634c141d573","Type":"ContainerDied","Data":"f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06"} Nov 01 04:01:16.310475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.310368 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v" event={"ID":"ba0f64b8-e0f8-485d-9e23-2634c141d573","Type":"ContainerDied","Data":"186f34b6f55aae59cfcea025bf7c8d54349b5fdb3fb70d3e943b937aced1c75b"} Nov 01 04:01:16.310475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.310387 6735 scope.go:122] "RemoveContainer" containerID="f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06" Nov 01 04:01:16.320332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.320309 6735 scope.go:122] "RemoveContainer" containerID="f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06" Nov 01 04:01:16.320597 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:01:16.320579 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06\": container with ID starting with f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06 not found: ID does not exist" containerID="f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06" Nov 01 04:01:16.320645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.320607 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06"} err="failed to get container status \"f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06\": rpc error: code = NotFound desc = could not find container \"f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06\": container with ID starting with f64211cae9c9e2f5049219be5d3fd0291babc4535caa1b3ad57ad591776f9f06 not found: ID does not exist" Nov 01 04:01:16.465379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.465347 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v"] Nov 01 04:01:16.469267 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.469241 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/e2e-conflict-b-9e9797da-kserve-845b4977db-lt28v"] Nov 01 04:01:16.526089 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:16.526056 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0f64b8-e0f8-485d-9e23-2634c141d573" path="/var/lib/kubelet/pods/ba0f64b8-e0f8-485d-9e23-2634c141d573/volumes" Nov 01 04:01:17.260600 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:17.260563 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" podUID="00e7b1ed-7d66-4baa-989c-4656597cd10c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:01:18.146367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:18.146337 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb"] Nov 01 04:01:18.147853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:18.147824 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" podUID="00e7b1ed-7d66-4baa-989c-4656597cd10c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:01:18.192507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:18.192480 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8"] Nov 01 04:01:23.147369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.147330 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" podUID="00e7b1ed-7d66-4baa-989c-4656597cd10c" containerName="payload-processing" containerID="cri-o://c9f5c604bbdba4738f95571fd6d6f58419a7ac2b15654b5599964518cce90d51" gracePeriod=25 Nov 01 04:01:23.192975 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.192934 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" podUID="2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" containerName="payload-processing" containerID="cri-o://77132ff0388ba70e7c5a13ca2c2ac1dd3d93856bce03388ec38ea9f50d80e491" gracePeriod=25 Nov 01 04:01:23.364206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.360265 6735 generic.go:356] "Generic (PLEG): container finished" podID="2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" containerID="77132ff0388ba70e7c5a13ca2c2ac1dd3d93856bce03388ec38ea9f50d80e491" exitCode=0 Nov 01 04:01:23.364206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.360335 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" event={"ID":"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7","Type":"ContainerDied","Data":"77132ff0388ba70e7c5a13ca2c2ac1dd3d93856bce03388ec38ea9f50d80e491"} Nov 01 04:01:23.364206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.362745 6735 generic.go:356] "Generic (PLEG): container finished" podID="00e7b1ed-7d66-4baa-989c-4656597cd10c" containerID="c9f5c604bbdba4738f95571fd6d6f58419a7ac2b15654b5599964518cce90d51" exitCode=0 Nov 01 04:01:23.364206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.362796 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" event={"ID":"00e7b1ed-7d66-4baa-989c-4656597cd10c","Type":"ContainerDied","Data":"c9f5c604bbdba4738f95571fd6d6f58419a7ac2b15654b5599964518cce90d51"} Nov 01 04:01:23.459981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.459954 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:23.545318 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.545288 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:01:23.558877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.558852 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-provider-credentials\") pod \"00e7b1ed-7d66-4baa-989c-4656597cd10c\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " Nov 01 04:01:23.559013 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.558894 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-routing-overlay-volume\") pod \"00e7b1ed-7d66-4baa-989c-4656597cd10c\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " Nov 01 04:01:23.559092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.559009 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-kube-api-access-cgsjd\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-kube-api-access-cgsjd\") pod \"00e7b1ed-7d66-4baa-989c-4656597cd10c\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " Nov 01 04:01:23.559156 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.559142 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-plugins-config-volume\") pod \"00e7b1ed-7d66-4baa-989c-4656597cd10c\" (UID: \"00e7b1ed-7d66-4baa-989c-4656597cd10c\") " Nov 01 04:01:23.559281 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.559260 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-routing-overlay-volume" pod "00e7b1ed-7d66-4baa-989c-4656597cd10c" (UID: "00e7b1ed-7d66-4baa-989c-4656597cd10c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:01:23.559530 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.559505 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-plugins-config-volume" pod "00e7b1ed-7d66-4baa-989c-4656597cd10c" (UID: "00e7b1ed-7d66-4baa-989c-4656597cd10c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:01:23.559805 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.559566 6735 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-routing-overlay-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:23.561164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.561143 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-provider-credentials" pod "00e7b1ed-7d66-4baa-989c-4656597cd10c" (UID: "00e7b1ed-7d66-4baa-989c-4656597cd10c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:01:23.561246 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.561208 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-kube-api-access-cgsjd" pod "00e7b1ed-7d66-4baa-989c-4656597cd10c" (UID: "00e7b1ed-7d66-4baa-989c-4656597cd10c"). InnerVolumeSpecName "kube-api-access-cgsjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:01:23.660706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.660665 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-kube-api-access-wlvd7\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-kube-api-access-wlvd7\") pod \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " Nov 01 04:01:23.660903 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.660825 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-routing-overlay-volume\") pod \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " Nov 01 04:01:23.660968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.660903 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-plugins-config-volume\") pod \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " Nov 01 04:01:23.660968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.660938 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-provider-credentials\") pod \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\" (UID: \"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7\") " Nov 01 04:01:23.661304 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.661277 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-routing-overlay-volume" pod "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" (UID: "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:01:23.661383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.661321 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-plugins-config-volume" pod "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" (UID: "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:01:23.661502 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.661484 6735 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-routing-overlay-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:23.661563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.661507 6735 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-provider-credentials\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:23.661563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.661522 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgsjd\" (UniqueName: \"kubernetes.io/projected/00e7b1ed-7d66-4baa-989c-4656597cd10c-kube-api-access-cgsjd\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:23.661563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.661537 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:23.661563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.661551 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e7b1ed-7d66-4baa-989c-4656597cd10c-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:23.662968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.662947 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-kube-api-access-wlvd7" pod "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" (UID: "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7"). InnerVolumeSpecName "kube-api-access-wlvd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:01:23.663128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.663107 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-provider-credentials" pod "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" (UID: "2d9ee1f3-bd87-43fe-8f45-c35ecac742f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:01:23.762265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.762224 6735 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-provider-credentials\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:23.762265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:23.762258 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvd7\" (UniqueName: \"kubernetes.io/projected/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7-kube-api-access-wlvd7\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:01:24.369786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.369761 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" Nov 01 04:01:24.370250 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.369805 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb" event={"ID":"00e7b1ed-7d66-4baa-989c-4656597cd10c","Type":"ContainerDied","Data":"6bcc4fe5850c9d8f534d45e4954c94a39d48e45bd54605c9f53a19d5594d568a"} Nov 01 04:01:24.370250 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.369871 6735 scope.go:122] "RemoveContainer" containerID="c9f5c604bbdba4738f95571fd6d6f58419a7ac2b15654b5599964518cce90d51" Nov 01 04:01:24.371277 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.371259 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" Nov 01 04:01:24.371337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.371291 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8" event={"ID":"2d9ee1f3-bd87-43fe-8f45-c35ecac742f7","Type":"ContainerDied","Data":"aa7a9b8f2a8d7a4a0d103c6535b951287dff988143c5bd0e877d3870fbfc0a44"} Nov 01 04:01:24.381248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.380760 6735 scope.go:122] "RemoveContainer" containerID="77132ff0388ba70e7c5a13ca2c2ac1dd3d93856bce03388ec38ea9f50d80e491" Nov 01 04:01:24.526688 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.526652 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8"] Nov 01 04:01:24.529275 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.529243 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-757c658c7b-pxlp8"] Nov 01 04:01:24.611480 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.611447 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:01:24.631350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.631307 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb"] Nov 01 04:01:24.636300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:24.636276 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6698d5675d-tkzwb"] Nov 01 04:01:26.570446 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:26.570409 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e7b1ed-7d66-4baa-989c-4656597cd10c" path="/var/lib/kubelet/pods/00e7b1ed-7d66-4baa-989c-4656597cd10c/volumes" Nov 01 04:01:26.571638 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:26.571612 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" path="/var/lib/kubelet/pods/2d9ee1f3-bd87-43fe-8f45-c35ecac742f7/volumes" Nov 01 04:01:33.001592 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:01:33.001556 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:01:44.564000 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.563967 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64"] Nov 01 04:01:44.564539 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564523 6735 cpu_manager.go:414] "containerMap: removing container" podUID="00e7b1ed-7d66-4baa-989c-4656597cd10c" containerName="payload-processing" Nov 01 04:01:44.564609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564541 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e7b1ed-7d66-4baa-989c-4656597cd10c" containerName="payload-processing" Nov 01 04:01:44.564609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564559 6735 cpu_manager.go:414] "containerMap: removing container" podUID="2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" containerName="payload-processing" Nov 01 04:01:44.564609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564564 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" containerName="payload-processing" Nov 01 04:01:44.564609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564592 6735 cpu_manager.go:414] "containerMap: removing container" podUID="ba0f64b8-e0f8-485d-9e23-2634c141d573" containerName="main" Nov 01 04:01:44.564609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564598 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0f64b8-e0f8-485d-9e23-2634c141d573" containerName="main" Nov 01 04:01:44.564759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564681 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0f64b8-e0f8-485d-9e23-2634c141d573" containerName="main" Nov 01 04:01:44.564759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564692 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e7b1ed-7d66-4baa-989c-4656597cd10c" containerName="payload-processing" Nov 01 04:01:44.564759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.564699 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9ee1f3-bd87-43fe-8f45-c35ecac742f7" containerName="payload-processing" Nov 01 04:01:44.569811 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.569794 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.610584 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.610553 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6"] Nov 01 04:01:44.614440 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.614422 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.662600 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662569 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.662774 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662609 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.662774 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662722 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.662774 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662752 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.662944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662786 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.662997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662942 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5tn\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-kube-api-access-bk5tn\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.662997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662966 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.662997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.662982 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.663167 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.663001 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.732961 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6"] Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.776728 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64"] Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788116 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788178 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a15781-c068-4965-87da-93bbaa57cc01-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788243 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqt7\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-kube-api-access-hfqt7\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788295 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5tn\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-kube-api-access-bk5tn\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788328 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788356 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788386 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788412 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788449 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788494 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788527 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788563 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788600 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788635 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788685 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788720 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a15781-c068-4965-87da-93bbaa57cc01-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788750 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.788794 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.795206 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.805448 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.805746 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.806360 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.806630 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.827550 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.830093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.827687 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.852844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.852810 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5tn\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-kube-api-access-bk5tn\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.865946 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.865913 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.884044 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.884014 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:44.889611 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889581 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.889755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889629 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.889755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889648 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.889755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889667 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.889755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889695 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a15781-c068-4965-87da-93bbaa57cc01-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.889755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889738 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.890082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889763 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a15781-c068-4965-87da-93bbaa57cc01-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.890082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889797 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqt7\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-kube-api-access-hfqt7\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.890082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.889844 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.890082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.890017 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-credential-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.890336 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.890317 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-socket\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.890879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.890838 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-certs\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.891077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.890926 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a15781-c068-4965-87da-93bbaa57cc01-istiod-ca-cert\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.891204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.891135 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-data\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.892867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.892836 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-envoy\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.893093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.893070 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a15781-c068-4965-87da-93bbaa57cc01-istio-podinfo\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.963204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.963020 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-istio-token\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:44.966069 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:44.965132 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqt7\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-kube-api-access-hfqt7\") pod \"e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:45.229951 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.229902 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:45.389877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.389844 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64"] Nov 01 04:01:45.425562 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.425523 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:45.425690 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.425613 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:45.425690 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.425656 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:45.618343 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.618305 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6"] Nov 01 04:01:45.629497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.629448 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:45.629582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.629555 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:45.629617 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:45.629600 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:46.074060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.073229 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" event={"ID":"9f336ac6-ee8b-4bc8-963f-e1da031cb351","Type":"ContainerStarted","Data":"f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7"} Nov 01 04:01:46.074060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.073297 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" event={"ID":"9f336ac6-ee8b-4bc8-963f-e1da031cb351","Type":"ContainerStarted","Data":"2ccb9f9f107fe14fdaa021ac331b3367ca1145bd8bb83b7d9220f5ebfc851c6f"} Nov 01 04:01:46.086978 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.076573 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" event={"ID":"c6a15781-c068-4965-87da-93bbaa57cc01","Type":"ContainerStarted","Data":"024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d"} Nov 01 04:01:46.086978 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.076635 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" event={"ID":"c6a15781-c068-4965-87da-93bbaa57cc01","Type":"ContainerStarted","Data":"c9699ca085d3b999a91490fd1520be2d7f019f6c27a75a938c70ee7dc3f8d279"} Nov 01 04:01:46.195656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.195586 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podStartSLOduration=2.195563618 podStartE2EDuration="2.195563618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:46.192162552 +0000 UTC m=+1117.781257192" watchObservedRunningTime="2026-11-01 04:01:46.195563618 +0000 UTC m=+1117.784658251" Nov 01 04:01:46.231060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.231000 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:46.232740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.232712 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 04:01:46.232855 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.232759 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 04:01:46.306864 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.306812 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podStartSLOduration=2.306796859 podStartE2EDuration="2.306796859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:46.304762189 +0000 UTC m=+1117.893856818" watchObservedRunningTime="2026-11-01 04:01:46.306796859 +0000 UTC m=+1117.895891539" Nov 01 04:01:46.892376 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.892341 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:46.894092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.893027 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 04:01:46.894343 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:46.894290 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 04:01:47.230995 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.230959 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 04:01:47.231220 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.231014 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 04:01:47.819532 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.819496 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97"] Nov 01 04:01:47.823729 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.823709 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.838627 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.838602 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.838751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.838678 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.838802 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.838782 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.838839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.838815 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.838876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.838836 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.838941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.838928 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.838977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.838953 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7bm\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-kube-api-access-hk7bm\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.839050 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.839021 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.839107 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.839096 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.890060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.889424 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 04:01:47.890060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.889484 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 04:01:47.923507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.923472 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97"] Nov 01 04:01:47.940569 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940536 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940618 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940672 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940691 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940707 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940974 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940757 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940974 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940772 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7bm\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-kube-api-access-hk7bm\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940974 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940807 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.940974 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.940824 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.941214 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.941031 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.941336 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.941313 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.941466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.941446 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.941891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.941730 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.941891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.941866 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.943544 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.943524 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:47.943678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:47.943662 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:48.004352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.004090 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7bm\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-kube-api-access-hk7bm\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:48.004352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.004322 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:48.138010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.137933 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:48.231189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.231152 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 04:01:48.231383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.231214 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 04:01:48.620919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.620895 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97"] Nov 01 04:01:48.623922 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:01:48.623897 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bc5c87_ce74_4d5b_9f18_37a314a6a483.slice/crio-0dc95c7d26b8c1a0f24ef8af9b4cca1865b428bf980bf383d276e29b82fe9609 WatchSource:0}: Error finding container 0dc95c7d26b8c1a0f24ef8af9b4cca1865b428bf980bf383d276e29b82fe9609: Status 404 returned error can't find the container with id 0dc95c7d26b8c1a0f24ef8af9b4cca1865b428bf980bf383d276e29b82fe9609 Nov 01 04:01:48.626605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.626568 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:48.626687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.626635 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:48.626687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.626666 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:01:48.886525 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.886432 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 04:01:48.886830 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:48.886773 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 04:01:49.109696 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.109646 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" event={"ID":"74bc5c87-ce74-4d5b-9f18-37a314a6a483","Type":"ContainerStarted","Data":"f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882"} Nov 01 04:01:49.110235 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.109710 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" event={"ID":"74bc5c87-ce74-4d5b-9f18-37a314a6a483","Type":"ContainerStarted","Data":"0dc95c7d26b8c1a0f24ef8af9b4cca1865b428bf980bf383d276e29b82fe9609"} Nov 01 04:01:49.139143 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.139078 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:49.140946 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.140923 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:49.141078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.140971 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:49.230938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.230883 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 04:01:49.231161 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.230947 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 04:01:49.233918 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.233865 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podStartSLOduration=2.233850496 podStartE2EDuration="2.233850496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:49.231538475 +0000 UTC m=+1120.820633095" watchObservedRunningTime="2026-11-01 04:01:49.233850496 +0000 UTC m=+1120.822945125" Nov 01 04:01:49.888272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.887812 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 04:01:49.888272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:49.887873 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 04:01:50.138998 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:50.138915 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:50.138998 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:50.138969 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:50.230871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:50.230817 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 04:01:50.231078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:50.230883 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 04:01:50.890114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:50.889569 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 04:01:50.890114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:50.889649 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 04:01:51.139193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:51.139162 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:51.139592 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:51.139215 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:51.230568 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:51.230533 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 04:01:51.230750 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:51.230583 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 04:01:51.897924 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:51.897470 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 01 04:01:51.897924 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:51.897534 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 01 04:01:52.138651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:52.138618 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:52.138820 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:52.138670 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:52.231116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:52.231086 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 01 04:01:52.231492 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:52.231140 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 01 04:01:53.138911 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.138867 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:53.139144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.138940 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:53.293284 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.293247 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:53.294446 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.294417 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:53.295609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.295592 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:01:53.509272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.507122 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:53.509272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.508070 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:53.509272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:53.509145 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:01:54.071028 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:54.070995 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:01:54.138366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:54.138336 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:54.138549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:54.138393 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:55.143063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:55.138398 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:55.143063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:55.138457 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:56.138443 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:56.138406 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 01 04:01:56.159628 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:56.159588 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 01 04:01:58.180229 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:58.180187 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:01:58.180874 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:58.180833 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:01:58.403076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:58.403028 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:58.403531 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:58.403501 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:01:58.404563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:01:58.404544 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:02:02.919114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:02.919071 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm"] Nov 01 04:02:02.923427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:02.923406 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:02.972009 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:02.971975 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"openshift-service-ca.crt\"" Nov 01 04:02:03.005958 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.005922 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-home\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.006179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.005976 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89dad693-8ce5-4348-a34c-0a3fb36e979e-tls-certs\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.006179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.006146 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6pgg\" (UniqueName: \"kubernetes.io/projected/89dad693-8ce5-4348-a34c-0a3fb36e979e-kube-api-access-t6pgg\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.006299 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.006273 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-model-cache\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.006344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.006313 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-tmp-dir\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.006387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.006372 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-dshm\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.009686 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:02:03.009654 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:02:03.018696 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.018659 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"default-dockercfg-2zg92\"" Nov 01 04:02:03.018839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.018702 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"distinct-2-w0-162191-kserve-self-signed-certs\"" Nov 01 04:02:03.018839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.018757 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-162191\"/\"kube-root-ca.crt\"" Nov 01 04:02:03.021679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.021657 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm"] Nov 01 04:02:03.111895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.111854 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-home\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.112719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.112688 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89dad693-8ce5-4348-a34c-0a3fb36e979e-tls-certs\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.113048 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.112612 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-home\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.113553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.113258 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6pgg\" (UniqueName: \"kubernetes.io/projected/89dad693-8ce5-4348-a34c-0a3fb36e979e-kube-api-access-t6pgg\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.113819 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.113802 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-model-cache\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.114295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.114277 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-tmp-dir\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.114797 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.114213 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-model-cache\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.114797 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.114661 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-tmp-dir\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.115007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.114990 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-dshm\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.116732 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.116584 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89dad693-8ce5-4348-a34c-0a3fb36e979e-tls-certs\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.123849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.123794 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-dshm\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.172934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.172852 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6pgg\" (UniqueName: \"kubernetes.io/projected/89dad693-8ce5-4348-a34c-0a3fb36e979e-kube-api-access-t6pgg\") pod \"distinct-2-w0-162191-kserve-84d4f78c4b-r88hm\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.236761 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.236731 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:03.717170 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:03.717142 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm"] Nov 01 04:02:03.719109 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:02:03.719081 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89dad693_8ce5_4348_a34c_0a3fb36e979e.slice/crio-370f723d946a56fdc65328532ae8963d45bff95793922975067980f4d881179d WatchSource:0}: Error finding container 370f723d946a56fdc65328532ae8963d45bff95793922975067980f4d881179d: Status 404 returned error can't find the container with id 370f723d946a56fdc65328532ae8963d45bff95793922975067980f4d881179d Nov 01 04:02:04.395953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:04.395919 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" event={"ID":"89dad693-8ce5-4348-a34c-0a3fb36e979e","Type":"ContainerStarted","Data":"2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d"} Nov 01 04:02:04.395953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:04.395962 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" event={"ID":"89dad693-8ce5-4348-a34c-0a3fb36e979e","Type":"ContainerStarted","Data":"370f723d946a56fdc65328532ae8963d45bff95793922975067980f4d881179d"} Nov 01 04:02:04.508372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:04.508319 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" podStartSLOduration=2.508305327 podStartE2EDuration="2.508305327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:04.505927718 +0000 UTC m=+1136.095022348" watchObservedRunningTime="2026-11-01 04:02:04.508305327 +0000 UTC m=+1136.097399956" Nov 01 04:02:05.237857 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:05.237822 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:11.075232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:11.075174 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:02:11.075919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:11.075886 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:02:13.237274 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:13.237236 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:13.250325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:13.250301 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:13.529466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:13.529377 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:02:28.984072 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:28.984015 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:02:33.000933 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:02:33.000897 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:02:33.184454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.184421 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl"] Nov 01 04:02:33.188683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.188666 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353305 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353213 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpchh\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-kube-api-access-cpchh\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353305 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353264 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353537 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353344 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353537 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353449 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353537 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353486 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353638 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353578 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353641 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79d751-1e54-43b5-8068-61207d04bfcc-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353681 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.353762 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.353730 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.384905 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.384870 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl"] Nov 01 04:02:33.455384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455354 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455441 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpchh\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-kube-api-access-cpchh\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455471 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455491 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455555 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455586 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455628 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455701 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79d751-1e54-43b5-8068-61207d04bfcc-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455738 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455790 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-credential-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.455945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.455839 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-socket\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.456148 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.456028 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-data\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.456148 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.456098 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-certs\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.456422 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.456402 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79d751-1e54-43b5-8068-61207d04bfcc-istiod-ca-cert\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.457997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.457975 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-envoy\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.458283 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.458266 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-podinfo\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.512899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.512860 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-token\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.513027 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.512943 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpchh\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-kube-api-access-cpchh\") pod \"e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:33.802375 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:33.802321 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:34.220919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.220888 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl"] Nov 01 04:02:34.222407 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:02:34.222379 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c79d751_1e54_43b5_8068_61207d04bfcc.slice/crio-df32a8fa3e5ead1ad0816f26742283878879a463d9dc4f91c219d0be150847e5 WatchSource:0}: Error finding container df32a8fa3e5ead1ad0816f26742283878879a463d9dc4f91c219d0be150847e5: Status 404 returned error can't find the container with id df32a8fa3e5ead1ad0816f26742283878879a463d9dc4f91c219d0be150847e5 Nov 01 04:02:34.224758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.224707 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:02:34.224838 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.224772 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:02:34.224838 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.224800 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:02:34.727522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.727480 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" event={"ID":"6c79d751-1e54-43b5-8068-61207d04bfcc","Type":"ContainerStarted","Data":"c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5"} Nov 01 04:02:34.727730 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.727540 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" event={"ID":"6c79d751-1e54-43b5-8068-61207d04bfcc","Type":"ContainerStarted","Data":"df32a8fa3e5ead1ad0816f26742283878879a463d9dc4f91c219d0be150847e5"} Nov 01 04:02:34.802760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.802728 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:34.843842 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:34.843783 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podStartSLOduration=1.843768732 podStartE2EDuration="1.843768732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:34.841903089 +0000 UTC m=+1166.430997717" watchObservedRunningTime="2026-11-01 04:02:34.843768732 +0000 UTC m=+1166.432863360" Nov 01 04:02:35.805077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:35.805028 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 04:02:35.805512 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:35.805090 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 04:02:36.803103 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:36.803056 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 04:02:36.803326 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:36.803164 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 04:02:37.802683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:37.802647 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 04:02:37.803130 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:37.802701 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 04:02:38.802977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:38.802943 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 04:02:38.803408 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:38.803007 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 04:02:39.802963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:39.802934 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 04:02:39.803176 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:39.802994 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 04:02:40.803280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:40.803243 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 04:02:40.803786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:40.803307 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 04:02:41.806847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:41.806819 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:41.807344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:41.807239 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:41.808175 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:41.808148 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:02:54.344468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.344434 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww"] Nov 01 04:02:54.348548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.348528 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.439064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.417437 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"kube-root-ca.crt\"" Nov 01 04:02:54.439064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.417534 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"openshift-service-ca.crt\"" Nov 01 04:02:54.463179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.462853 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tls-certs\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.463179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.462953 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-dshm\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.463179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.463022 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ck6s\" (UniqueName: \"kubernetes.io/projected/b7b2dbe6-6923-4d46-82db-6037b289c9d0-kube-api-access-4ck6s\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.463179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.463103 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-model-cache\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.463536 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.463346 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tmp-dir\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.463536 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.463415 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-home\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.469116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.468632 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"distinct-2-w3-f73b9f-kserve-self-signed-certs\"" Nov 01 04:02:54.473285 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.473197 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww"] Nov 01 04:02:54.473654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.473619 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f73b9f\"/\"default-dockercfg-qwscg\"" Nov 01 04:02:54.564366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564337 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tls-certs\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.564559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564408 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-dshm\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.564559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564456 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ck6s\" (UniqueName: \"kubernetes.io/projected/b7b2dbe6-6923-4d46-82db-6037b289c9d0-kube-api-access-4ck6s\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.564559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564501 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-model-cache\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.564734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564606 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tmp-dir\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.564734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564645 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-home\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.564961 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564928 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-model-cache\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.565027 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.564996 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-home\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.565103 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.565079 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tmp-dir\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.566948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.566922 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-dshm\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.567145 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.567130 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tls-certs\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.655561 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.655482 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ck6s\" (UniqueName: \"kubernetes.io/projected/b7b2dbe6-6923-4d46-82db-6037b289c9d0-kube-api-access-4ck6s\") pod \"distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:54.662187 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:54.662157 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:02:55.103264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:55.102664 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww"] Nov 01 04:02:55.105173 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:02:55.105145 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7b2dbe6_6923_4d46_82db_6037b289c9d0.slice/crio-008d7220ab18c57bc1a0905204cfae8ee49115fc58dc65291e7b1ab6098d0b80 WatchSource:0}: Error finding container 008d7220ab18c57bc1a0905204cfae8ee49115fc58dc65291e7b1ab6098d0b80: Status 404 returned error can't find the container with id 008d7220ab18c57bc1a0905204cfae8ee49115fc58dc65291e7b1ab6098d0b80 Nov 01 04:02:55.139307 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:55.139277 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" event={"ID":"b7b2dbe6-6923-4d46-82db-6037b289c9d0","Type":"ContainerStarted","Data":"008d7220ab18c57bc1a0905204cfae8ee49115fc58dc65291e7b1ab6098d0b80"} Nov 01 04:02:56.147228 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:56.147191 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" event={"ID":"b7b2dbe6-6923-4d46-82db-6037b289c9d0","Type":"ContainerStarted","Data":"f45d786929070002ff6f3cb412b10e94090876c0b705fd098dc080486f6d597b"} Nov 01 04:02:56.260104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:56.260050 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" podStartSLOduration=2.260019224 podStartE2EDuration="2.260019224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:56.258451632 +0000 UTC m=+1187.847546261" watchObservedRunningTime="2026-11-01 04:02:56.260019224 +0000 UTC m=+1187.849113854" Nov 01 04:02:56.663007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:02:56.662971 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:03:03.001586 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:03:03.001548 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:03:04.663159 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:04.663127 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:03:04.675863 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:04.675835 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:03:04.989899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:04.989863 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:03:05.298438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:05.298358 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:03:07.271062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:07.270222 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:07.271595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:07.271145 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:07.299061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:07.297052 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:07.525088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:07.524958 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" podUID="89dad693-8ce5-4348-a34c-0a3fb36e979e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:08.009615 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.009568 6735 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Nov 01 04:03:08.315387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.288121 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 04:03:08.315387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.288500 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:03:08.315387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.288532 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:08.315387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.289806 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:03:08.315387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.314390 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:08.315387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.315187 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 04:03:08.787076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:08.778384 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 04:03:09.148116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.095766 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:03:09.148116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.095821 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:09.317292 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.317236 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:09.320810 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.320777 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:03:09.325094 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.321022 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:03:09.524396 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.524340 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:03:09.524617 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.524434 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:03:09.607454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.607416 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:03:09.608512 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:09.608493 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:03:18.671521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.671484 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w"] Nov 01 04:03:18.677219 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.677191 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:18.838062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.778951 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-078e0d65\"" Nov 01 04:03:18.838062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.787268 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-078e0d65-dockercfg-nmkzf\"" Nov 01 04:03:18.864056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.838906 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w"] Nov 01 04:03:18.884067 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.872762 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5sz4\" (UniqueName: \"kubernetes.io/projected/f8939b2b-f043-4768-a14a-69b7dde1ecde-kube-api-access-p5sz4\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:18.884453 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.884412 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8939b2b-f043-4768-a14a-69b7dde1ecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:18.973894 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.973861 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm"] Nov 01 04:03:18.979557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.979518 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm"] Nov 01 04:03:18.979688 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.979627 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:18.994977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.994949 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8939b2b-f043-4768-a14a-69b7dde1ecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:18.995139 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.995120 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5sz4\" (UniqueName: \"kubernetes.io/projected/f8939b2b-f043-4768-a14a-69b7dde1ecde-kube-api-access-p5sz4\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:18.995862 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:18.995838 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8939b2b-f043-4768-a14a-69b7dde1ecde-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:19.047978 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.047937 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4"] Nov 01 04:03:19.052023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.052006 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.055312 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.055287 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5sz4\" (UniqueName: \"kubernetes.io/projected/f8939b2b-f043-4768-a14a-69b7dde1ecde-kube-api-access-p5sz4\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:19.096622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.096586 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2fk\" (UniqueName: \"kubernetes.io/projected/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-kube-api-access-zd2fk\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.096787 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.096650 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-plugins-config-volume\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.111608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.111582 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4"] Nov 01 04:03:19.198286 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198259 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198302 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198434 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czmb\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-kube-api-access-7czmb\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198580 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198511 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198580 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198536 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198580 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198573 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198729 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198694 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2fk\" (UniqueName: \"kubernetes.io/projected/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-kube-api-access-zd2fk\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.198780 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198726 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198780 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198752 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-plugins-config-volume\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.198916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198784 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.198916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.198811 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.199563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.199545 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-plugins-config-volume\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.258057 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.257970 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2fk\" (UniqueName: \"kubernetes.io/projected/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-kube-api-access-zd2fk\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.290790 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.290762 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:19.297019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.296999 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.299608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.299580 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.299722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.299639 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.299722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.299679 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.299722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.299710 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.299899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.299842 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.299962 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.299906 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.300109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.300021 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czmb\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-kube-api-access-7czmb\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.300169 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.300121 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-certs\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.300169 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.300145 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.300337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.300176 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.300642 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.300407 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.300642 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.300523 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-data\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.300800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.300733 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istiod-ca-cert\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.301079 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.301009 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-credential-socket\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.302404 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.302380 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-podinfo\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.303019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.302995 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-envoy\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.373789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.360253 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czmb\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-kube-api-access-7czmb\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.373789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.370304 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-token\") pod \"e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.377472 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.377426 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:19.667737 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.667668 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w"] Nov 01 04:03:19.669408 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:03:19.669372 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8939b2b_f043_4768_a14a_69b7dde1ecde.slice/crio-7471a6717b2a537b8fe9b275e4aa1832ac7d3ff89294a923664fe99e2a0fc30a WatchSource:0}: Error finding container 7471a6717b2a537b8fe9b275e4aa1832ac7d3ff89294a923664fe99e2a0fc30a: Status 404 returned error can't find the container with id 7471a6717b2a537b8fe9b275e4aa1832ac7d3ff89294a923664fe99e2a0fc30a Nov 01 04:03:19.715603 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.715577 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm"] Nov 01 04:03:19.716228 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:03:19.716196 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda724a5_56d1_4cf5_9b6a_bf5a08fb609e.slice/crio-b278b01e147fc08d28a884b3e062d488125377af1849328fb9c56cd5e491504f WatchSource:0}: Error finding container b278b01e147fc08d28a884b3e062d488125377af1849328fb9c56cd5e491504f: Status 404 returned error can't find the container with id b278b01e147fc08d28a884b3e062d488125377af1849328fb9c56cd5e491504f Nov 01 04:03:19.771568 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:03:19.771535 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdcf4ef_8793_44c3_8b13_a6e76e0cdb9d.slice/crio-26c930e97ff8fcd1cda19a6e7e215c73192950cc51f04f1bdc4a72e667e8b507 WatchSource:0}: Error finding container 26c930e97ff8fcd1cda19a6e7e215c73192950cc51f04f1bdc4a72e667e8b507: Status 404 returned error can't find the container with id 26c930e97ff8fcd1cda19a6e7e215c73192950cc51f04f1bdc4a72e667e8b507 Nov 01 04:03:19.771967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.771942 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4"] Nov 01 04:03:19.774264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.774237 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:03:19.774350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.774298 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:03:19.774350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.774325 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:03:19.978065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.978011 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" event={"ID":"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d","Type":"ContainerStarted","Data":"b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b"} Nov 01 04:03:19.978279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.978095 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" event={"ID":"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d","Type":"ContainerStarted","Data":"26c930e97ff8fcd1cda19a6e7e215c73192950cc51f04f1bdc4a72e667e8b507"} Nov 01 04:03:19.980026 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.979995 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" event={"ID":"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e","Type":"ContainerStarted","Data":"96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f"} Nov 01 04:03:19.980193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.980067 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" event={"ID":"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e","Type":"ContainerStarted","Data":"b278b01e147fc08d28a884b3e062d488125377af1849328fb9c56cd5e491504f"} Nov 01 04:03:19.980193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.980094 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:19.981772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.981749 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" event={"ID":"f8939b2b-f043-4768-a14a-69b7dde1ecde","Type":"ContainerStarted","Data":"67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e"} Nov 01 04:03:19.981891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.981786 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" event={"ID":"f8939b2b-f043-4768-a14a-69b7dde1ecde","Type":"ContainerStarted","Data":"7471a6717b2a537b8fe9b275e4aa1832ac7d3ff89294a923664fe99e2a0fc30a"} Nov 01 04:03:19.981891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:19.981815 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:20.094174 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:20.094116 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podStartSLOduration=1.094102573 podStartE2EDuration="1.094102573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:20.091544216 +0000 UTC m=+1211.680638845" watchObservedRunningTime="2026-11-01 04:03:20.094102573 +0000 UTC m=+1211.683197193" Nov 01 04:03:20.206056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:20.202271 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" podStartSLOduration=2.202252498 podStartE2EDuration="2.202252498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:20.200547535 +0000 UTC m=+1211.789642165" watchObservedRunningTime="2026-11-01 04:03:20.202252498 +0000 UTC m=+1211.791347127" Nov 01 04:03:20.320120 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:20.319399 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" podStartSLOduration=2.319378584 podStartE2EDuration="2.319378584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:20.316342194 +0000 UTC m=+1211.905436824" watchObservedRunningTime="2026-11-01 04:03:20.319378584 +0000 UTC m=+1211.908473216" Nov 01 04:03:20.391062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:20.387385 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:20.399061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:20.396235 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 04:03:20.399061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:20.396315 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 04:03:21.382367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:21.382333 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 04:03:21.382787 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:21.382385 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 04:03:22.383207 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:22.383175 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 04:03:22.383600 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:22.383228 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 04:03:23.388189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:23.387094 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 04:03:23.388189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:23.387156 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 04:03:24.382579 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:24.382546 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 04:03:24.382764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:24.382600 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 04:03:25.382947 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:25.382913 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 04:03:25.383366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:25.382967 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 04:03:26.382869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:26.382836 6735 patch_prober.go:29] interesting pod/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 01 04:03:26.383141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:26.382902 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 01 04:03:27.386718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:27.386689 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:27.387248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:27.387094 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:27.387672 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:27.387651 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:03:29.627060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.616459 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp"] Nov 01 04:03:29.627060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.621502 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:29.650010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.649634 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp"] Nov 01 04:03:29.694948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.691369 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-182e51\"" Nov 01 04:03:29.694948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.691527 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-182e51-dockercfg-lvtnc\"" Nov 01 04:03:29.765060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.762663 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79"] Nov 01 04:03:29.788460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.788420 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:29.789092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.788931 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zldhl\" (UniqueName: \"kubernetes.io/projected/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-kube-api-access-zldhl\") pod \"payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:29.791339 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.791316 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:29.846883 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.845975 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79"] Nov 01 04:03:29.890700 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.890604 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zldhl\" (UniqueName: \"kubernetes.io/projected/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-kube-api-access-zldhl\") pod \"payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:29.890700 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.890693 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:29.890940 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.890884 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqj6v\" (UniqueName: \"kubernetes.io/projected/ae096853-319c-473e-962e-4c0a1f94eaf3-kube-api-access-gqj6v\") pod \"payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:29.890940 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.890926 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae096853-319c-473e-962e-4c0a1f94eaf3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:29.891640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.891617 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:29.948520 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.948483 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zldhl\" (UniqueName: \"kubernetes.io/projected/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-kube-api-access-zldhl\") pod \"payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:29.992599 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.992569 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqj6v\" (UniqueName: \"kubernetes.io/projected/ae096853-319c-473e-962e-4c0a1f94eaf3-kube-api-access-gqj6v\") pod \"payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:29.992806 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.992614 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae096853-319c-473e-962e-4c0a1f94eaf3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:29.993361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:29.993341 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae096853-319c-473e-962e-4c0a1f94eaf3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:30.050352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:30.050313 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqj6v\" (UniqueName: \"kubernetes.io/projected/ae096853-319c-473e-962e-4c0a1f94eaf3-kube-api-access-gqj6v\") pod \"payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:30.178340 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:30.178241 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:30.242451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:30.242421 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:30.646067 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:30.642479 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79"] Nov 01 04:03:30.752110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:30.752084 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp"] Nov 01 04:03:30.753764 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:03:30.753733 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2889b1_7ca4_4b1d_821f_1ccd1f9435d2.slice/crio-f1fafd605a7060eadaae47c5543af1b297f2e775c12c3b94170fe683bdc7ce86 WatchSource:0}: Error finding container f1fafd605a7060eadaae47c5543af1b297f2e775c12c3b94170fe683bdc7ce86: Status 404 returned error can't find the container with id f1fafd605a7060eadaae47c5543af1b297f2e775c12c3b94170fe683bdc7ce86 Nov 01 04:03:30.991136 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:30.991104 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:03:30.991288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:30.991177 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:03:31.117783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.117742 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" event={"ID":"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2","Type":"ContainerStarted","Data":"d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0"} Nov 01 04:03:31.117983 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.117823 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:31.117983 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.117847 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" event={"ID":"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2","Type":"ContainerStarted","Data":"f1fafd605a7060eadaae47c5543af1b297f2e775c12c3b94170fe683bdc7ce86"} Nov 01 04:03:31.120202 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.120160 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" event={"ID":"ae096853-319c-473e-962e-4c0a1f94eaf3","Type":"ContainerStarted","Data":"ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780"} Nov 01 04:03:31.120328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.120226 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" event={"ID":"ae096853-319c-473e-962e-4c0a1f94eaf3","Type":"ContainerStarted","Data":"d28692b768f8e172ff16ed365f29701ff6d3a56b51bf9b4df401578c62092733"} Nov 01 04:03:31.120328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.120287 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:31.321453 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.321341 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" podStartSLOduration=2.321327717 podStartE2EDuration="2.321327717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:31.320296816 +0000 UTC m=+1222.909391445" watchObservedRunningTime="2026-11-01 04:03:31.321327717 +0000 UTC m=+1222.910422346" Nov 01 04:03:31.438565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:31.438500 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" podStartSLOduration=2.438479989 podStartE2EDuration="2.438479989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:31.436465856 +0000 UTC m=+1223.025560487" watchObservedRunningTime="2026-11-01 04:03:31.438479989 +0000 UTC m=+1223.027574619" Nov 01 04:03:33.019440 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:03:33.019393 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:03:42.129594 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:42.129566 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:03:42.130110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:42.129638 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:03:56.655449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:03:56.655418 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:04:03.001830 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:04:03.001795 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:04:04.598314 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.598279 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw"] Nov 01 04:04:04.603210 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.603188 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.652199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.652170 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-182e51-serving-cert\"" Nov 01 04:04:04.685255 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.685220 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-metrics-certs\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.685448 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.685329 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgshm\" (UniqueName: \"kubernetes.io/projected/85375448-d138-4843-bf33-fe52eb296595-kube-api-access-mgshm\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.685517 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.685475 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.685577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.685513 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85375448-d138-4843-bf33-fe52eb296595-openshift-service-ca\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.701132 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.701093 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw"] Nov 01 04:04:04.787145 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.787109 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-metrics-certs\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.787311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.787191 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgshm\" (UniqueName: \"kubernetes.io/projected/85375448-d138-4843-bf33-fe52eb296595-kube-api-access-mgshm\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.787311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.787289 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.787424 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.787313 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85375448-d138-4843-bf33-fe52eb296595-openshift-service-ca\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.787479 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:04:04.787451 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-serving-cert: secret "maas-api-e2e-api-a-182e51-serving-cert" not found Nov 01 04:04:04.787552 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:04:04.787537 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls podName:85375448-d138-4843-bf33-fe52eb296595 nodeName:}" failed. No retries permitted until 2026-11-01 04:04:05.287516352 +0000 UTC m=+1256.876610973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls") pod "maas-api-e2e-api-a-182e51-698b867859-6v4nw" (UID: "85375448-d138-4843-bf33-fe52eb296595") : secret "maas-api-e2e-api-a-182e51-serving-cert" not found Nov 01 04:04:04.787938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.787920 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85375448-d138-4843-bf33-fe52eb296595-openshift-service-ca\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.789727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.789705 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-metrics-certs\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:04.904524 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:04.904060 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgshm\" (UniqueName: \"kubernetes.io/projected/85375448-d138-4843-bf33-fe52eb296595-kube-api-access-mgshm\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:05.341060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.325861 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:05.341377 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.327084 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr"] Nov 01 04:04:05.366087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.357274 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls\") pod \"maas-api-e2e-api-a-182e51-698b867859-6v4nw\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:05.398660 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.398632 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.454656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.454569 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr"] Nov 01 04:04:05.464431 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.464400 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-aab51f-openshift-default-dockercfg-9jmww\"" Nov 01 04:04:05.518017 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.517984 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:05.556853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.556821 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-token\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.556932 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-envoy\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.556981 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-socket\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557125 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.557073 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-credential-socket\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.557169 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-certs\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557241 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.557220 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwfp\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-kube-api-access-pbwfp\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.557359 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istiod-ca-cert\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.557467 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-podinfo\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.557572 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.557554 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-data\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660025 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-podinfo\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660137 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-data\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660177 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-token\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660266 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-envoy\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660325 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-socket\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660428 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-credential-socket\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660482 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-certs\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660521 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwfp\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-kube-api-access-pbwfp\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.660630 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istiod-ca-cert\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.667887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.661612 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istiod-ca-cert\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.676663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.675887 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-credential-socket\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.676663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.676188 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-data\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.696730 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.696689 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-certs\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.697684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.697653 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-socket\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.718384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.718337 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-envoy\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.723372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.723299 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-podinfo\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.734533 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.734467 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-token\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.768071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.768023 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwfp\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-kube-api-access-pbwfp\") pod \"e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.793205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.793170 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:05.902890 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:04:05.902857 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85375448_d138_4843_bf33_fe52eb296595.slice/crio-fa92180122c754ec4b9b747399ddb4f1f2e0300104dbf333cd03c09ec5372cb0 WatchSource:0}: Error finding container fa92180122c754ec4b9b747399ddb4f1f2e0300104dbf333cd03c09ec5372cb0: Status 404 returned error can't find the container with id fa92180122c754ec4b9b747399ddb4f1f2e0300104dbf333cd03c09ec5372cb0 Nov 01 04:04:05.903167 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:05.903149 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw"] Nov 01 04:04:06.167960 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.167931 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr"] Nov 01 04:04:06.169557 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:04:06.169525 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c10c88c_389b_41b7_be99_a1dd5655fd4e.slice/crio-cfafbdd2a2d81f77641911cbc5578ad4ccb95a9a058865b1f7faf433923ed18c WatchSource:0}: Error finding container cfafbdd2a2d81f77641911cbc5578ad4ccb95a9a058865b1f7faf433923ed18c: Status 404 returned error can't find the container with id cfafbdd2a2d81f77641911cbc5578ad4ccb95a9a058865b1f7faf433923ed18c Nov 01 04:04:06.172097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.172016 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:06.172163 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.172140 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:06.172200 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.172173 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:06.501304 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.501267 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" event={"ID":"1c10c88c-389b-41b7-be99-a1dd5655fd4e","Type":"ContainerStarted","Data":"91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda"} Nov 01 04:04:06.501506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.501322 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" event={"ID":"1c10c88c-389b-41b7-be99-a1dd5655fd4e","Type":"ContainerStarted","Data":"cfafbdd2a2d81f77641911cbc5578ad4ccb95a9a058865b1f7faf433923ed18c"} Nov 01 04:04:06.502719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.502692 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" event={"ID":"85375448-d138-4843-bf33-fe52eb296595","Type":"ContainerStarted","Data":"fa92180122c754ec4b9b747399ddb4f1f2e0300104dbf333cd03c09ec5372cb0"} Nov 01 04:04:06.620649 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.620587 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" podStartSLOduration=1.620573602 podStartE2EDuration="1.620573602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:04:06.617487421 +0000 UTC m=+1258.206582049" watchObservedRunningTime="2026-11-01 04:04:06.620573602 +0000 UTC m=+1258.209668302" Nov 01 04:04:06.793341 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:06.793319 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:07.390210 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.390177 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5"] Nov 01 04:04:07.394432 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.394409 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.443540 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.443513 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-865066-openshift-default-dockercfg-97xzq\"" Nov 01 04:04:07.493392 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.493354 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5"] Nov 01 04:04:07.500508 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500479 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.500508 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500511 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.500758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500537 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.500758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500600 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893283b0-538c-4124-95ee-5a36ddc405f5-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.500758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500677 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.500758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500712 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893283b0-538c-4124-95ee-5a36ddc405f5-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.500758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500756 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.501025 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500772 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt464\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-kube-api-access-bt464\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.501025 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.500829 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.514477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.514438 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" event={"ID":"85375448-d138-4843-bf33-fe52eb296595","Type":"ContainerStarted","Data":"ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956"} Nov 01 04:04:07.514686 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.514671 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:07.601689 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.601656 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.601689 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.601692 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.601928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.601719 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.601928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.601748 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893283b0-538c-4124-95ee-5a36ddc405f5-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.601928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.601793 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.601928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.601854 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893283b0-538c-4124-95ee-5a36ddc405f5-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.601989 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.602026 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt464\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-kube-api-access-bt464\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.602097 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.602116 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-data\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.602180 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-credential-socket\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.602220 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.602658 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893283b0-538c-4124-95ee-5a36ddc405f5-istiod-ca-cert\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.602985 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.602894 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-certs\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.604557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.604537 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-envoy\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.604648 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.604631 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893283b0-538c-4124-95ee-5a36ddc405f5-istio-podinfo\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.627330 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.627276 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" podStartSLOduration=2.747246679 podStartE2EDuration="3.627262051s" totalImagesPullingTime="880.015372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:04:07.625121413 +0000 UTC m=+1259.214216043" watchObservedRunningTime="2026-11-01 04:04:07.627262051 +0000 UTC m=+1259.216356741" Nov 01 04:04:07.659765 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.659696 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-istio-token\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.659948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.659925 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt464\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-kube-api-access-bt464\") pod \"e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.707484 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.707450 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:07.865666 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:07.865384 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:08.259617 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.259585 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5"] Nov 01 04:04:08.261089 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:04:08.261056 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893283b0_538c_4124_95ee_5a36ddc405f5.slice/crio-c559a665f1ab34be0d6a342815ac1bcd23691561e3426dd2f77ebfc2857c2e99 WatchSource:0}: Error finding container c559a665f1ab34be0d6a342815ac1bcd23691561e3426dd2f77ebfc2857c2e99: Status 404 returned error can't find the container with id c559a665f1ab34be0d6a342815ac1bcd23691561e3426dd2f77ebfc2857c2e99 Nov 01 04:04:08.263634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.263600 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:08.263707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.263668 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:08.263707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.263696 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:08.529354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.529268 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" event={"ID":"893283b0-538c-4124-95ee-5a36ddc405f5","Type":"ContainerStarted","Data":"a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603"} Nov 01 04:04:08.529354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.529327 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" event={"ID":"893283b0-538c-4124-95ee-5a36ddc405f5","Type":"ContainerStarted","Data":"c559a665f1ab34be0d6a342815ac1bcd23691561e3426dd2f77ebfc2857c2e99"} Nov 01 04:04:08.529354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.529353 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:08.529565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.529404 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:04:08.643810 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.643661 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podStartSLOduration=1.643647412 podStartE2EDuration="1.643647412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:04:08.639562802 +0000 UTC m=+1260.228657431" watchObservedRunningTime="2026-11-01 04:04:08.643647412 +0000 UTC m=+1260.232742041" Nov 01 04:04:08.708645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:08.708591 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:09.710108 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:09.710071 6735 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 04:04:09.710489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:09.710124 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 04:04:10.708661 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:10.708624 6735 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 04:04:10.708813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:10.708679 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 04:04:11.708926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:11.708877 6735 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 04:04:11.709542 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:11.708949 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 04:04:12.708481 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:12.708434 6735 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 04:04:12.708666 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:12.708506 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 04:04:13.532542 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:13.532512 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:04:13.708608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:13.708573 6735 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 04:04:13.708779 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:13.708638 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 04:04:14.708663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:14.708571 6735 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 04:04:14.709151 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:14.708686 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 04:04:15.796484 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:15.796448 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:15.797273 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:15.797245 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:15.798257 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:15.798236 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:04:33.002082 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:04:33.002029 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:04:49.440543 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:49.440510 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:04:50.916713 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:50.916676 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc"] Nov 01 04:04:50.921204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:50.921181 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:50.969828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:50.969809 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-aab51f-serving-cert\"" Nov 01 04:04:51.002596 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.002563 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfbv\" (UniqueName: \"kubernetes.io/projected/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-kube-api-access-lnfbv\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.002767 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.002702 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-openshift-service-ca\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.002837 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.002787 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-metrics-certs\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.002837 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.002820 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.019133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.019105 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc"] Nov 01 04:04:51.104452 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.104403 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-metrics-certs\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.104452 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.104464 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.104671 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:04:51.104593 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-serving-cert: secret "maas-api-e2e-api-a-aab51f-serving-cert" not found Nov 01 04:04:51.104671 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:04:51.104663 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls podName:10e5bcb3-80b0-4aa0-a000-0ab53b6cc276 nodeName:}" failed. No retries permitted until 2026-11-01 04:04:51.604641028 +0000 UTC m=+1303.193735639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls") pod "maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" (UID: "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276") : secret "maas-api-e2e-api-a-aab51f-serving-cert" not found Nov 01 04:04:51.104747 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.104681 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfbv\" (UniqueName: \"kubernetes.io/projected/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-kube-api-access-lnfbv\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.104781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.104761 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-openshift-service-ca\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.105811 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.105779 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-openshift-service-ca\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.107022 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.107004 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-metrics-certs\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.210116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.205723 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfbv\" (UniqueName: \"kubernetes.io/projected/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-kube-api-access-lnfbv\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.641526 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.641416 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.644129 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.644108 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls\") pod \"maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:51.834321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:51.834288 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:52.205400 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:52.205378 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc"] Nov 01 04:04:52.935463 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:52.935432 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" event={"ID":"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276","Type":"ContainerStarted","Data":"e6eae3ef4e0506366c556a346c64059f8f5bea667662091da4f64c56f8bac151"} Nov 01 04:04:53.723067 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.723001 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr"] Nov 01 04:04:53.727498 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.727477 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.786062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.784400 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3344e8-openshift-default-dockercfg-5jmgg\"" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797515 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-certs\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797565 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797612 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-token\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797662 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-credential-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797741 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-data\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797779 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-envoy\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797874 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-podinfo\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.797958 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnkh\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-kube-api-access-9tnkh\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.803063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.798093 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istiod-ca-cert\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.867063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.858502 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr"] Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.937890 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnkh\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-kube-api-access-9tnkh\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938109 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istiod-ca-cert\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938156 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-certs\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938182 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938227 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-token\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938263 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-credential-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938337 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-data\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938372 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-envoy\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.941082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.938469 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-podinfo\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.955056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.953690 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.955056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.954659 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istiod-ca-cert\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.955056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.954955 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-certs\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.955332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.955212 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-credential-socket\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.958625 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.955691 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-data\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.969911 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.964140 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-podinfo\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:53.987122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.981667 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" event={"ID":"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276","Type":"ContainerStarted","Data":"d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2"} Nov 01 04:04:53.987122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:53.982930 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:04:54.000815 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:54.000778 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-envoy\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:54.059057 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:54.055279 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-token\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:54.104105 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:54.102801 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnkh\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-kube-api-access-9tnkh\") pod \"e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:54.124850 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:54.124786 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" podStartSLOduration=3.145761579 podStartE2EDuration="4.124765176s" totalImagesPullingTime="979.003597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:04:54.122341925 +0000 UTC m=+1305.711436568" watchObservedRunningTime="2026-11-01 04:04:54.124765176 +0000 UTC m=+1305.713859807" Nov 01 04:04:54.366755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:54.366668 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:54.996979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:54.996947 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr"] Nov 01 04:04:54.997805 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:04:54.997784 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47420ed3_26f3_4302_ad0d_2e41e9a94a7c.slice/crio-aeb3400e270a754fc669f992969ccae3ed3363a8666033194531cc9f45ca971f WatchSource:0}: Error finding container aeb3400e270a754fc669f992969ccae3ed3363a8666033194531cc9f45ca971f: Status 404 returned error can't find the container with id aeb3400e270a754fc669f992969ccae3ed3363a8666033194531cc9f45ca971f Nov 01 04:04:55.000232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:55.000196 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:55.000320 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:55.000279 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:55.000320 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:55.000306 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:04:55.996343 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:55.996301 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" event={"ID":"47420ed3-26f3-4302-ad0d-2e41e9a94a7c","Type":"ContainerStarted","Data":"ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e"} Nov 01 04:04:55.996552 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:55.996358 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" event={"ID":"47420ed3-26f3-4302-ad0d-2e41e9a94a7c","Type":"ContainerStarted","Data":"aeb3400e270a754fc669f992969ccae3ed3363a8666033194531cc9f45ca971f"} Nov 01 04:04:56.158721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:56.158671 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podStartSLOduration=3.158656603 podStartE2EDuration="3.158656603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:04:56.156516817 +0000 UTC m=+1307.745611448" watchObservedRunningTime="2026-11-01 04:04:56.158656603 +0000 UTC m=+1307.747751296" Nov 01 04:04:56.370116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:56.369062 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:56.384348 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:56.384315 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:57.002004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:57.001974 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:04:57.002803 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:04:57.002784 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:05:01.001732 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:01.001701 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:05:03.001781 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:05:03.001744 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:05:17.273827 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:17.273775 6735 generic.go:356] "Generic (PLEG): container finished" podID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerID="91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda" exitCode=0 Nov 01 04:05:17.274491 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:17.273995 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" event={"ID":"1c10c88c-389b-41b7-be99-a1dd5655fd4e","Type":"ContainerDied","Data":"91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda"} Nov 01 04:05:17.274561 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:17.274524 6735 scope.go:122] "RemoveContainer" containerID="91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda" Nov 01 04:05:17.283144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:17.282777 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:05:17.283144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:17.282892 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:05:17.283144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:17.282947 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:05:18.281892 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:18.281854 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" event={"ID":"1c10c88c-389b-41b7-be99-a1dd5655fd4e","Type":"ContainerStarted","Data":"8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f"} Nov 01 04:05:18.794319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:18.794265 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:05:18.795141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:18.795112 6735 patch_prober.go:29] interesting pod/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 04:05:18.795277 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:18.795173 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 04:05:19.793781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:19.793746 6735 patch_prober.go:29] interesting pod/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 04:05:19.794262 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:19.793809 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 04:05:20.793521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:20.793472 6735 patch_prober.go:29] interesting pod/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 04:05:20.793705 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:20.793551 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 04:05:21.793568 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:21.793538 6735 patch_prober.go:29] interesting pod/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 04:05:21.793963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:21.793590 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 04:05:22.793932 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:22.793898 6735 patch_prober.go:29] interesting pod/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 04:05:22.794398 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:22.793950 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 04:05:23.526860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:23.526832 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:05:23.798355 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:23.798276 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:05:24.330919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:24.330892 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:05:32.561492 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:32.561458 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:05:33.001156 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:05:33.001125 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:05:39.124616 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.124580 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv"] Nov 01 04:05:39.127991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.127971 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.151914 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.151876 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-credential-socket\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152117 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.151944 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-token\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.152111 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istiod-ca-cert\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.152153 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-envoy\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.152173 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbrv\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-kube-api-access-4jbrv\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.152207 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-data\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.152277 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-podinfo\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.152323 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-socket\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.152349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.152344 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-certs\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.177306 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.177274 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-78dad9-openshift-default-dockercfg-j5ptt\"" Nov 01 04:05:39.231076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.228375 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv"] Nov 01 04:05:39.253283 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253252 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-socket\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253483 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253298 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-certs\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253483 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253367 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-credential-socket\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253483 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253410 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-token\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253500 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istiod-ca-cert\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253546 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-envoy\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253577 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbrv\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-kube-api-access-4jbrv\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253630 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-data\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253661 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-socket\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253711 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-certs\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253803 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-podinfo\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.253964 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.253947 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-credential-socket\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.254175 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.254157 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-data\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.254385 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.254367 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istiod-ca-cert\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.255937 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.255912 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-envoy\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.256344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.256323 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-podinfo\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.381126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.356812 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-token\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.396833 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.396801 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbrv\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-kube-api-access-4jbrv\") pod \"e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.442369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.442309 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:39.952899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.952869 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv"] Nov 01 04:05:39.954582 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:05:39.954542 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1380b1a8_c7ec_448b_ba69_4e4321c073a7.slice/crio-3bdaeb63f4e31b16589d1bc49f737495ac246e2f7b7de4563cab74984eecbcd4 WatchSource:0}: Error finding container 3bdaeb63f4e31b16589d1bc49f737495ac246e2f7b7de4563cab74984eecbcd4: Status 404 returned error can't find the container with id 3bdaeb63f4e31b16589d1bc49f737495ac246e2f7b7de4563cab74984eecbcd4 Nov 01 04:05:39.957103 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.957066 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:05:39.957176 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.957140 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:05:39.957221 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:39.957177 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:05:40.490024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:40.489993 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" event={"ID":"1380b1a8-c7ec-448b-ba69-4e4321c073a7","Type":"ContainerStarted","Data":"ef1e1d37adfe2e4019f3a3696231588e2088b3275988d50113c02607bc52a9a2"} Nov 01 04:05:40.490422 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:40.490072 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" event={"ID":"1380b1a8-c7ec-448b-ba69-4e4321c073a7","Type":"ContainerStarted","Data":"3bdaeb63f4e31b16589d1bc49f737495ac246e2f7b7de4563cab74984eecbcd4"} Nov 01 04:05:40.604800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:40.604748 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podStartSLOduration=1.604730434 podStartE2EDuration="1.604730434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:40.603451879 +0000 UTC m=+1352.192546509" watchObservedRunningTime="2026-11-01 04:05:40.604730434 +0000 UTC m=+1352.193825113" Nov 01 04:05:41.447072 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:41.444431 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:41.464213 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:41.462350 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:41.540062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:41.537092 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:41.563073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:41.560350 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:05:47.543801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.543767 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8"] Nov 01 04:05:47.551935 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.551912 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.605029 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.604995 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8"] Nov 01 04:05:47.606946 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.606918 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8562485e-8985-4042-a0c2-1b0c1f0193d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.607370 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.607344 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.607510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.607486 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-metrics-certs\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.607604 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.607585 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9sc5\" (UniqueName: \"kubernetes.io/projected/8562485e-8985-4042-a0c2-1b0c1f0193d9-kube-api-access-w9sc5\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.647247 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.647219 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3344e8-serving-cert\"" Nov 01 04:05:47.727159 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.727118 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.732082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.727618 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-metrics-certs\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.732427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.732400 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9sc5\" (UniqueName: \"kubernetes.io/projected/8562485e-8985-4042-a0c2-1b0c1f0193d9-kube-api-access-w9sc5\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.732709 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.732692 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8562485e-8985-4042-a0c2-1b0c1f0193d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.740068 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:05:47.727497 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-serving-cert: secret "maas-api-e2e-api-b-3344e8-serving-cert" not found Nov 01 04:05:47.740068 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:05:47.737481 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls podName:8562485e-8985-4042-a0c2-1b0c1f0193d9 nodeName:}" failed. No retries permitted until 2026-11-01 04:05:48.237456574 +0000 UTC m=+1359.826551183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls") pod "maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" (UID: "8562485e-8985-4042-a0c2-1b0c1f0193d9") : secret "maas-api-e2e-api-b-3344e8-serving-cert" not found Nov 01 04:05:47.741013 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.740984 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8562485e-8985-4042-a0c2-1b0c1f0193d9-openshift-service-ca\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:47.780486 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.780451 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-metrics-certs\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:48.008065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:47.986354 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7"] Nov 01 04:05:48.011922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.011865 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.025816 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.025771 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7"] Nov 01 04:05:48.056525 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.048981 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644cq\" (UniqueName: \"kubernetes.io/projected/4ddb652f-13af-4a31-86d0-331f531e8ccc-kube-api-access-644cq\") pod \"payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.056525 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.049278 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddb652f-13af-4a31-86d0-331f531e8ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.068642 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.067313 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-78dad9-dockercfg-psdn8\"" Nov 01 04:05:48.083080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.074296 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-78dad9\"" Nov 01 04:05:48.206257 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.206149 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644cq\" (UniqueName: \"kubernetes.io/projected/4ddb652f-13af-4a31-86d0-331f531e8ccc-kube-api-access-644cq\") pod \"payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.210059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.206454 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddb652f-13af-4a31-86d0-331f531e8ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.221061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.218589 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddb652f-13af-4a31-86d0-331f531e8ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.242593 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.237705 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9sc5\" (UniqueName: \"kubernetes.io/projected/8562485e-8985-4042-a0c2-1b0c1f0193d9-kube-api-access-w9sc5\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:48.289190 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.283423 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-644cq\" (UniqueName: \"kubernetes.io/projected/4ddb652f-13af-4a31-86d0-331f531e8ccc-kube-api-access-644cq\") pod \"payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.317068 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.312249 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:48.420806 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.420772 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls\") pod \"maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:48.476944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.476914 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:48.500903 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.500874 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:48.905123 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.905006 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7"] Nov 01 04:05:48.908656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.908542 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8"] Nov 01 04:05:48.953407 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:05:48.953381 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8562485e_8985_4042_a0c2_1b0c1f0193d9.slice/crio-e77e41e94b866e3a885c97265603e693476bedfcc94f1dfb8dfd05fbf2d91ff5 WatchSource:0}: Error finding container e77e41e94b866e3a885c97265603e693476bedfcc94f1dfb8dfd05fbf2d91ff5: Status 404 returned error can't find the container with id e77e41e94b866e3a885c97265603e693476bedfcc94f1dfb8dfd05fbf2d91ff5 Nov 01 04:05:48.954989 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:48.954967 6735 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 04:05:49.588561 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:49.588518 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" event={"ID":"8562485e-8985-4042-a0c2-1b0c1f0193d9","Type":"ContainerStarted","Data":"e77e41e94b866e3a885c97265603e693476bedfcc94f1dfb8dfd05fbf2d91ff5"} Nov 01 04:05:49.590328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:49.590295 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" event={"ID":"4ddb652f-13af-4a31-86d0-331f531e8ccc","Type":"ContainerStarted","Data":"4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8"} Nov 01 04:05:49.590518 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:49.590350 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" event={"ID":"4ddb652f-13af-4a31-86d0-331f531e8ccc","Type":"ContainerStarted","Data":"3e6f48a6effbb14dfb0893a331c7468ef677203b976a3cb44efde8297d238e21"} Nov 01 04:05:49.590518 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:49.590456 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:05:49.699203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:49.699143 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" podStartSLOduration=2.699123286 podStartE2EDuration="2.699123286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:49.698357797 +0000 UTC m=+1361.287452427" watchObservedRunningTime="2026-11-01 04:05:49.699123286 +0000 UTC m=+1361.288217916" Nov 01 04:05:50.599342 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:50.599306 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" event={"ID":"8562485e-8985-4042-a0c2-1b0c1f0193d9","Type":"ContainerStarted","Data":"c244467a0f52da67d45a2b96074c0c96bcd901f98d4d5d5764c36bc12f092bd3"} Nov 01 04:05:50.599917 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:50.599700 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:50.710382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:50.710328 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" podStartSLOduration=2.624411819 podStartE2EDuration="3.710314829s" totalImagesPullingTime="1.08590301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:50.709366572 +0000 UTC m=+1362.298461200" watchObservedRunningTime="2026-11-01 04:05:50.710314829 +0000 UTC m=+1362.299409457" Nov 01 04:05:56.611775 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:56.611749 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:05:57.471059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.467404 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl"] Nov 01 04:05:57.504383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.504355 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:57.573644 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.573602 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl"] Nov 01 04:05:57.599198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.574825 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6dd36b-dockercfg-dxnhd\"" Nov 01 04:05:57.599393 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.574952 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6dd36b\"" Nov 01 04:05:57.639819 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.639787 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ca7817-a2c1-49b2-84b7-3777f0206fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:57.640333 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.640315 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwz4r\" (UniqueName: \"kubernetes.io/projected/c3ca7817-a2c1-49b2-84b7-3777f0206fae-kube-api-access-xwz4r\") pod \"payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:57.741951 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.741877 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwz4r\" (UniqueName: \"kubernetes.io/projected/c3ca7817-a2c1-49b2-84b7-3777f0206fae-kube-api-access-xwz4r\") pod \"payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:57.742168 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.742146 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ca7817-a2c1-49b2-84b7-3777f0206fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:57.742900 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.742879 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ca7817-a2c1-49b2-84b7-3777f0206fae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:57.799488 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.799460 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwz4r\" (UniqueName: \"kubernetes.io/projected/c3ca7817-a2c1-49b2-84b7-3777f0206fae-kube-api-access-xwz4r\") pod \"payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:57.918022 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:57.917995 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:58.296537 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:58.296509 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl"] Nov 01 04:05:58.297762 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:05:58.297735 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ca7817_a2c1_49b2_84b7_3777f0206fae.slice/crio-c6990deba7f1ec1254d25edba0254e51c81787fb154c1ea4da9826743fc50986 WatchSource:0}: Error finding container c6990deba7f1ec1254d25edba0254e51c81787fb154c1ea4da9826743fc50986: Status 404 returned error can't find the container with id c6990deba7f1ec1254d25edba0254e51c81787fb154c1ea4da9826743fc50986 Nov 01 04:05:58.697631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:58.697537 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" event={"ID":"c3ca7817-a2c1-49b2-84b7-3777f0206fae","Type":"ContainerStarted","Data":"aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b"} Nov 01 04:05:58.697631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:58.697594 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" event={"ID":"c3ca7817-a2c1-49b2-84b7-3777f0206fae","Type":"ContainerStarted","Data":"c6990deba7f1ec1254d25edba0254e51c81787fb154c1ea4da9826743fc50986"} Nov 01 04:05:58.698142 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:58.697652 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:05:58.849254 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:05:58.849198 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" podStartSLOduration=1.8491827299999999 podStartE2EDuration="1.84918273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:58.846118987 +0000 UTC m=+1370.435213617" watchObservedRunningTime="2026-11-01 04:05:58.84918273 +0000 UTC m=+1370.438277355" Nov 01 04:06:00.602808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:00.602778 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:06:03.002751 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:06:03.002715 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:06:06.703645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.703598 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp"] Nov 01 04:06:06.715303 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.715275 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp"] Nov 01 04:06:06.715458 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.715372 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:06.732575 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.732536 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:06.732752 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.732673 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gd8v\" (UniqueName: \"kubernetes.io/projected/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-kube-api-access-6gd8v\") pod \"payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:06.764679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.764637 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4adf4c\"" Nov 01 04:06:06.811010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.810984 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4adf4c-dockercfg-lgpq4\"" Nov 01 04:06:06.834207 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.834172 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:06.834364 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.834260 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gd8v\" (UniqueName: \"kubernetes.io/projected/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-kube-api-access-6gd8v\") pod \"payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:06.834853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.834834 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:06.892049 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:06.892015 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gd8v\" (UniqueName: \"kubernetes.io/projected/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-kube-api-access-6gd8v\") pod \"payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:07.030656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:07.030618 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:07.402825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:07.402792 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp"] Nov 01 04:06:07.405471 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:06:07.405438 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354fb55a_c27e_40b7_a8bd_0bd9e4cab222.slice/crio-acf93e68eb4d140af6e1ed4d3e89844430ff2d72d0f3092e7385bb27d21d8726 WatchSource:0}: Error finding container acf93e68eb4d140af6e1ed4d3e89844430ff2d72d0f3092e7385bb27d21d8726: Status 404 returned error can't find the container with id acf93e68eb4d140af6e1ed4d3e89844430ff2d72d0f3092e7385bb27d21d8726 Nov 01 04:06:07.760595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:07.760542 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" event={"ID":"354fb55a-c27e-40b7-a8bd-0bd9e4cab222","Type":"ContainerStarted","Data":"4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1"} Nov 01 04:06:07.761018 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:07.760735 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:07.761018 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:07.760781 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" event={"ID":"354fb55a-c27e-40b7-a8bd-0bd9e4cab222","Type":"ContainerStarted","Data":"acf93e68eb4d140af6e1ed4d3e89844430ff2d72d0f3092e7385bb27d21d8726"} Nov 01 04:06:07.870211 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:07.870142 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" podStartSLOduration=1.8701136109999998 podStartE2EDuration="1.870113611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:07.868725339 +0000 UTC m=+1379.457819969" watchObservedRunningTime="2026-11-01 04:06:07.870113611 +0000 UTC m=+1379.459208241" Nov 01 04:06:09.707640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:09.707607 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:06:17.424889 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:17.424853 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:06:18.780967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:18.780938 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:06:27.662567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.660993 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": read tcp 10.134.0.2:42570->10.134.0.88:15021: read: connection reset by peer" start-of-body= Nov 01 04:06:27.662567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.661078 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": read tcp 10.134.0.2:42570->10.134.0.88:15021: read: connection reset by peer" Nov 01 04:06:27.917609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.917515 6735 generic.go:356] "Generic (PLEG): container finished" podID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerID="ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e" exitCode=0 Nov 01 04:06:27.917609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.917567 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" event={"ID":"47420ed3-26f3-4302-ad0d-2e41e9a94a7c","Type":"ContainerDied","Data":"ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e"} Nov 01 04:06:27.917926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.917912 6735 scope.go:122] "RemoveContainer" containerID="ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e" Nov 01 04:06:27.919552 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.919519 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:06:27.919615 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.919596 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:06:27.919654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:27.919628 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:06:28.926395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:28.926355 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" event={"ID":"47420ed3-26f3-4302-ad0d-2e41e9a94a7c","Type":"ContainerStarted","Data":"37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574"} Nov 01 04:06:29.382073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:29.376082 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:06:29.382073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:29.376202 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 04:06:29.382073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:29.376246 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 04:06:30.368062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:30.367442 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 04:06:30.368062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:30.367508 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 04:06:31.369492 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:31.369449 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 04:06:31.370172 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:31.370133 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 04:06:32.368499 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:32.368464 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 04:06:32.368789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:32.368763 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 04:06:33.001450 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:06:33.001410 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:06:33.367513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:33.367419 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 04:06:33.367802 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:33.367761 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 04:06:34.372842 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:34.372808 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 04:06:34.373497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:34.373464 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 04:06:35.371859 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:35.371830 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:06:35.372274 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:35.372255 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:06:35.372906 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:35.372891 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:06:48.989558 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:48.989487 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5"] Nov 01 04:06:48.995364 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:48.995341 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.091670 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.091631 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6dd36b-serving-cert\"" Nov 01 04:06:49.095207 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.095179 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5"] Nov 01 04:06:49.149222 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.149190 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.149503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.149455 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/a4ae00aa-d7fa-4fc0-911a-2df51505c219-kube-api-access-mpwnp\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.149634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.149613 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-metrics-certs\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.149698 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.149651 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae00aa-d7fa-4fc0-911a-2df51505c219-openshift-service-ca\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.276363 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.276261 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/a4ae00aa-d7fa-4fc0-911a-2df51505c219-kube-api-access-mpwnp\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.282144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.282107 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-metrics-certs\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.311377 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.298814 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-metrics-certs\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.311636 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.311330 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae00aa-d7fa-4fc0-911a-2df51505c219-openshift-service-ca\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.312768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.312618 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae00aa-d7fa-4fc0-911a-2df51505c219-openshift-service-ca\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.313090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.313071 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.313468 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:06:49.313453 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-serving-cert: secret "maas-api-e2e-api-b-6dd36b-serving-cert" not found Nov 01 04:06:49.313647 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:06:49.313629 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls podName:a4ae00aa-d7fa-4fc0-911a-2df51505c219 nodeName:}" failed. No retries permitted until 2026-11-01 04:06:49.813606219 +0000 UTC m=+1421.402700839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls") pod "maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" (UID: "a4ae00aa-d7fa-4fc0-911a-2df51505c219") : secret "maas-api-e2e-api-b-6dd36b-serving-cert" not found Nov 01 04:06:49.352181 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.352135 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/a4ae00aa-d7fa-4fc0-911a-2df51505c219-kube-api-access-mpwnp\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.826095 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.826063 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.828620 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.828596 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls\") pod \"maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:49.909891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:49.909859 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:50.464059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:50.448101 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5"] Nov 01 04:06:50.517867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:50.517812 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:06:51.163264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:51.163195 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" event={"ID":"a4ae00aa-d7fa-4fc0-911a-2df51505c219","Type":"ContainerStarted","Data":"b33a8b569ccd4f50d61fa8f26d48fcc4e44c075c580aa064c2856e30b1cf60a8"} Nov 01 04:06:52.178158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:52.178112 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" event={"ID":"a4ae00aa-d7fa-4fc0-911a-2df51505c219","Type":"ContainerStarted","Data":"b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf"} Nov 01 04:06:52.178578 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:52.178183 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:06:52.295902 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:52.295833 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" podStartSLOduration=3.223803812 podStartE2EDuration="4.295819285s" totalImagesPullingTime="1.072015473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:52.293536188 +0000 UTC m=+1423.882630821" watchObservedRunningTime="2026-11-01 04:06:52.295819285 +0000 UTC m=+1423.884913914" Nov 01 04:06:58.199104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:06:58.199073 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:07:02.265765 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:02.265734 6735 generic.go:356] "Generic (PLEG): container finished" podID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerID="ef1e1d37adfe2e4019f3a3696231588e2088b3275988d50113c02607bc52a9a2" exitCode=0 Nov 01 04:07:02.266353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:02.265799 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" event={"ID":"1380b1a8-c7ec-448b-ba69-4e4321c073a7","Type":"ContainerDied","Data":"ef1e1d37adfe2e4019f3a3696231588e2088b3275988d50113c02607bc52a9a2"} Nov 01 04:07:02.266423 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:02.266377 6735 scope.go:122] "RemoveContainer" containerID="ef1e1d37adfe2e4019f3a3696231588e2088b3275988d50113c02607bc52a9a2" Nov 01 04:07:02.268085 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:02.268027 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:07:02.268164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:02.268125 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:07:02.268164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:02.268160 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:07:03.002313 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:07:03.002275 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:07:03.275645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:03.275548 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" event={"ID":"1380b1a8-c7ec-448b-ba69-4e4321c073a7","Type":"ContainerStarted","Data":"f07a8cc15d0e3649a917efbc92a2762fc68ddc9cab857e8cab63d9753fbe6719"} Nov 01 04:07:03.443507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:03.443478 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:07:03.443673 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:03.443651 6735 patch_prober.go:29] interesting pod/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 04:07:03.443725 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:03.443703 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 04:07:03.597534 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:03.597446 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm"] Nov 01 04:07:03.697011 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:03.696978 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w"] Nov 01 04:07:04.443953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:04.443923 6735 patch_prober.go:29] interesting pod/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 04:07:04.444389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:04.443986 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 04:07:05.565466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:05.565430 6735 patch_prober.go:29] interesting pod/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 04:07:05.566099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:05.566064 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 04:07:06.443853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:06.443816 6735 patch_prober.go:29] interesting pod/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 04:07:06.444061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:06.443872 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 04:07:07.443888 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:07.443856 6735 patch_prober.go:29] interesting pod/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 04:07:07.444320 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:07.443928 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 04:07:08.419910 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:08.419881 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79"] Nov 01 04:07:08.443133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:08.443100 6735 patch_prober.go:29] interesting pod/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 04:07:08.443300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:08.443170 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 04:07:08.533822 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:08.533788 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp"] Nov 01 04:07:08.603147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:08.603103 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" podUID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" containerName="payload-processing" containerID="cri-o://96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f" gracePeriod=25 Nov 01 04:07:08.704546 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:08.704448 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" podUID="f8939b2b-f043-4768-a14a-69b7dde1ecde" containerName="payload-pre-processing" containerID="cri-o://67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e" gracePeriod=25 Nov 01 04:07:09.443411 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:09.443375 6735 patch_prober.go:29] interesting pod/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 04:07:09.443598 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:09.443434 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 04:07:10.468416 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:10.468383 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:07:10.468941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:10.468924 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:07:10.469811 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:10.469788 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:07:10.989454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:10.989419 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" podUID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:07:10.989454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:10.989440 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" podUID="f8939b2b-f043-4768-a14a-69b7dde1ecde" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:07:13.420955 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.420881 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" podUID="ae096853-319c-473e-962e-4c0a1f94eaf3" containerName="payload-processing" containerID="cri-o://ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780" gracePeriod=25 Nov 01 04:07:13.538620 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.538568 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" podUID="fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" containerName="payload-pre-processing" containerID="cri-o://d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0" gracePeriod=25 Nov 01 04:07:13.726077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.724907 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:07:13.786463 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.786438 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae096853-319c-473e-962e-4c0a1f94eaf3-kube-api-access-gqj6v\" (UniqueName: \"kubernetes.io/projected/ae096853-319c-473e-962e-4c0a1f94eaf3-kube-api-access-gqj6v\") pod \"ae096853-319c-473e-962e-4c0a1f94eaf3\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " Nov 01 04:07:13.786587 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.786487 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae096853-319c-473e-962e-4c0a1f94eaf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae096853-319c-473e-962e-4c0a1f94eaf3-plugins-config-volume\") pod \"ae096853-319c-473e-962e-4c0a1f94eaf3\" (UID: \"ae096853-319c-473e-962e-4c0a1f94eaf3\") " Nov 01 04:07:13.786877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.786853 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae096853-319c-473e-962e-4c0a1f94eaf3-plugins-config-volume" pod "ae096853-319c-473e-962e-4c0a1f94eaf3" (UID: "ae096853-319c-473e-962e-4c0a1f94eaf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:07:13.787264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.787202 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae096853-319c-473e-962e-4c0a1f94eaf3-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:13.788693 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.788672 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae096853-319c-473e-962e-4c0a1f94eaf3-kube-api-access-gqj6v" pod "ae096853-319c-473e-962e-4c0a1f94eaf3" (UID: "ae096853-319c-473e-962e-4c0a1f94eaf3"). InnerVolumeSpecName "kube-api-access-gqj6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:07:13.807494 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.807464 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:07:13.888401 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.888365 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-plugins-config-volume\") pod \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " Nov 01 04:07:13.888570 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.888413 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-kube-api-access-zldhl\" (UniqueName: \"kubernetes.io/projected/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-kube-api-access-zldhl\") pod \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\" (UID: \"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2\") " Nov 01 04:07:13.888813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.888789 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-plugins-config-volume" pod "fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" (UID: "fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:07:13.889107 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.889082 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:13.889107 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.889104 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqj6v\" (UniqueName: \"kubernetes.io/projected/ae096853-319c-473e-962e-4c0a1f94eaf3-kube-api-access-gqj6v\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:13.890707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.890683 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-kube-api-access-zldhl" pod "fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" (UID: "fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2"). InnerVolumeSpecName "kube-api-access-zldhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:07:13.990485 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:13.990447 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zldhl\" (UniqueName: \"kubernetes.io/projected/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2-kube-api-access-zldhl\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:14.462489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.462459 6735 generic.go:356] "Generic (PLEG): container finished" podID="ae096853-319c-473e-962e-4c0a1f94eaf3" containerID="ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780" exitCode=0 Nov 01 04:07:14.462954 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.462514 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" Nov 01 04:07:14.462954 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.462535 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" event={"ID":"ae096853-319c-473e-962e-4c0a1f94eaf3","Type":"ContainerDied","Data":"ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780"} Nov 01 04:07:14.462954 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.462586 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79" event={"ID":"ae096853-319c-473e-962e-4c0a1f94eaf3","Type":"ContainerDied","Data":"d28692b768f8e172ff16ed365f29701ff6d3a56b51bf9b4df401578c62092733"} Nov 01 04:07:14.462954 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.462610 6735 scope.go:122] "RemoveContainer" containerID="ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780" Nov 01 04:07:14.464233 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.464209 6735 generic.go:356] "Generic (PLEG): container finished" podID="fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" containerID="d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0" exitCode=0 Nov 01 04:07:14.464353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.464246 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" Nov 01 04:07:14.464353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.464274 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" event={"ID":"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2","Type":"ContainerDied","Data":"d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0"} Nov 01 04:07:14.464353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.464306 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp" event={"ID":"fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2","Type":"ContainerDied","Data":"f1fafd605a7060eadaae47c5543af1b297f2e775c12c3b94170fe683bdc7ce86"} Nov 01 04:07:14.473265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.473165 6735 scope.go:122] "RemoveContainer" containerID="ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780" Nov 01 04:07:14.473533 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:07:14.473508 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780\": container with ID starting with ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780 not found: ID does not exist" containerID="ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780" Nov 01 04:07:14.473587 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.473539 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780"} err="failed to get container status \"ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780\": rpc error: code = NotFound desc = could not find container \"ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780\": container with ID starting with ad505e49d6c108a75c205042a39f43fffaf107a2e48e19828d7fa6e143a81780 not found: ID does not exist" Nov 01 04:07:14.473587 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.473557 6735 scope.go:122] "RemoveContainer" containerID="d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0" Nov 01 04:07:14.482987 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.482971 6735 scope.go:122] "RemoveContainer" containerID="d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0" Nov 01 04:07:14.483292 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:07:14.483274 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0\": container with ID starting with d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0 not found: ID does not exist" containerID="d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0" Nov 01 04:07:14.483353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.483299 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0"} err="failed to get container status \"d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0\": rpc error: code = NotFound desc = could not find container \"d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0\": container with ID starting with d5a1d5ed95186c98e479ded78cf0e89e0b4ce18cc54852d9377b81dd27eeace0 not found: ID does not exist" Nov 01 04:07:14.617389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.617359 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79"] Nov 01 04:07:14.664564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.664488 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-182e51-9b9bfcc67-qjg79"] Nov 01 04:07:14.722886 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.722852 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp"] Nov 01 04:07:14.728637 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:14.728612 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-182e51-7dc55f6df5-2s6gp"] Nov 01 04:07:16.527479 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:16.527445 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae096853-319c-473e-962e-4c0a1f94eaf3" path="/var/lib/kubelet/pods/ae096853-319c-473e-962e-4c0a1f94eaf3/volumes" Nov 01 04:07:16.527890 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:16.527876 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" path="/var/lib/kubelet/pods/fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2/volumes" Nov 01 04:07:20.989677 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:20.989633 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" podUID="f8939b2b-f043-4768-a14a-69b7dde1ecde" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:07:20.990100 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:20.989727 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" podUID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:07:28.873144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:28.873072 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:07:28.930147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:28.930117 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-plugins-config-volume\") pod \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " Nov 01 04:07:28.930313 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:28.930210 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-kube-api-access-zd2fk\" (UniqueName: \"kubernetes.io/projected/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-kube-api-access-zd2fk\") pod \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\" (UID: \"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e\") " Nov 01 04:07:28.930779 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:28.930574 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-plugins-config-volume" pod "cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" (UID: "cda724a5-56d1-4cf5-9b6a-bf5a08fb609e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:07:28.931344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:28.930895 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:28.932851 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:28.932808 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-kube-api-access-zd2fk" pod "cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" (UID: "cda724a5-56d1-4cf5-9b6a-bf5a08fb609e"). InnerVolumeSpecName "kube-api-access-zd2fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:07:28.996410 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:28.996379 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:07:29.031724 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.031681 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8939b2b-f043-4768-a14a-69b7dde1ecde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8939b2b-f043-4768-a14a-69b7dde1ecde-plugins-config-volume\") pod \"f8939b2b-f043-4768-a14a-69b7dde1ecde\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " Nov 01 04:07:29.031910 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.031789 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8939b2b-f043-4768-a14a-69b7dde1ecde-kube-api-access-p5sz4\" (UniqueName: \"kubernetes.io/projected/f8939b2b-f043-4768-a14a-69b7dde1ecde-kube-api-access-p5sz4\") pod \"f8939b2b-f043-4768-a14a-69b7dde1ecde\" (UID: \"f8939b2b-f043-4768-a14a-69b7dde1ecde\") " Nov 01 04:07:29.032178 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.032150 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8939b2b-f043-4768-a14a-69b7dde1ecde-plugins-config-volume" pod "f8939b2b-f043-4768-a14a-69b7dde1ecde" (UID: "f8939b2b-f043-4768-a14a-69b7dde1ecde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:07:29.032459 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.032426 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd2fk\" (UniqueName: \"kubernetes.io/projected/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e-kube-api-access-zd2fk\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:29.032459 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.032453 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8939b2b-f043-4768-a14a-69b7dde1ecde-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:29.034140 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.034108 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8939b2b-f043-4768-a14a-69b7dde1ecde-kube-api-access-p5sz4" pod "f8939b2b-f043-4768-a14a-69b7dde1ecde" (UID: "f8939b2b-f043-4768-a14a-69b7dde1ecde"). InnerVolumeSpecName "kube-api-access-p5sz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:07:29.134307 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.134221 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5sz4\" (UniqueName: \"kubernetes.io/projected/f8939b2b-f043-4768-a14a-69b7dde1ecde-kube-api-access-p5sz4\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:07:29.640061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.622188 6735 generic.go:356] "Generic (PLEG): container finished" podID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" containerID="96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f" exitCode=0 Nov 01 04:07:29.640061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.622293 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" event={"ID":"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e","Type":"ContainerDied","Data":"96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f"} Nov 01 04:07:29.640061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.622351 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" event={"ID":"cda724a5-56d1-4cf5-9b6a-bf5a08fb609e","Type":"ContainerDied","Data":"b278b01e147fc08d28a884b3e062d488125377af1849328fb9c56cd5e491504f"} Nov 01 04:07:29.640061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.622387 6735 scope.go:122] "RemoveContainer" containerID="96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f" Nov 01 04:07:29.640061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.622550 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm" Nov 01 04:07:29.752061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.750378 6735 generic.go:356] "Generic (PLEG): container finished" podID="f8939b2b-f043-4768-a14a-69b7dde1ecde" containerID="67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e" exitCode=0 Nov 01 04:07:29.752061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.750694 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" event={"ID":"f8939b2b-f043-4768-a14a-69b7dde1ecde","Type":"ContainerDied","Data":"67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e"} Nov 01 04:07:29.752061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.750716 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" Nov 01 04:07:29.752061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.750765 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w" event={"ID":"f8939b2b-f043-4768-a14a-69b7dde1ecde","Type":"ContainerDied","Data":"7471a6717b2a537b8fe9b275e4aa1832ac7d3ff89294a923664fe99e2a0fc30a"} Nov 01 04:07:29.764546 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.758193 6735 scope.go:122] "RemoveContainer" containerID="96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f" Nov 01 04:07:29.764546 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:07:29.758538 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f\": container with ID starting with 96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f not found: ID does not exist" containerID="96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f" Nov 01 04:07:29.764546 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.758567 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f"} err="failed to get container status \"96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f\": rpc error: code = NotFound desc = could not find container \"96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f\": container with ID starting with 96fe3577182ee2b36be1815e15d6210027790831b0aad69b46810d8a33e1144f not found: ID does not exist" Nov 01 04:07:29.764546 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.758590 6735 scope.go:122] "RemoveContainer" containerID="67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e" Nov 01 04:07:29.768767 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.768743 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:07:29.771126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.771110 6735 scope.go:122] "RemoveContainer" containerID="67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e" Nov 01 04:07:29.771483 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:07:29.771465 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e\": container with ID starting with 67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e not found: ID does not exist" containerID="67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e" Nov 01 04:07:29.771531 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.771493 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e"} err="failed to get container status \"67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e\": rpc error: code = NotFound desc = could not find container \"67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e\": container with ID starting with 67d46349640778ec9ee1f174b47973c73c08c974148c4745948258822b3af57e not found: ID does not exist" Nov 01 04:07:29.911167 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.911085 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm"] Nov 01 04:07:29.916977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:29.916953 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-crnqm"] Nov 01 04:07:30.019380 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:30.019344 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w"] Nov 01 04:07:30.022877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:30.022851 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-vww9w"] Nov 01 04:07:30.528198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:30.528164 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" path="/var/lib/kubelet/pods/cda724a5-56d1-4cf5-9b6a-bf5a08fb609e/volumes" Nov 01 04:07:30.528774 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:30.528755 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8939b2b-f043-4768-a14a-69b7dde1ecde" path="/var/lib/kubelet/pods/f8939b2b-f043-4768-a14a-69b7dde1ecde/volumes" Nov 01 04:07:33.001268 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:07:33.001228 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:07:50.428854 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.428800 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf"] Nov 01 04:07:50.430789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.430584 6735 cpu_manager.go:414] "containerMap: removing container" podUID="fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" containerName="payload-pre-processing" Nov 01 04:07:50.430975 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.430955 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" containerName="payload-pre-processing" Nov 01 04:07:50.431167 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.431151 6735 cpu_manager.go:414] "containerMap: removing container" podUID="ae096853-319c-473e-962e-4c0a1f94eaf3" containerName="payload-processing" Nov 01 04:07:50.431334 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.431321 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae096853-319c-473e-962e-4c0a1f94eaf3" containerName="payload-processing" Nov 01 04:07:50.431444 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.431434 6735 cpu_manager.go:414] "containerMap: removing container" podUID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" containerName="payload-processing" Nov 01 04:07:50.431521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.431511 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" containerName="payload-processing" Nov 01 04:07:50.431612 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.431602 6735 cpu_manager.go:414] "containerMap: removing container" podUID="f8939b2b-f043-4768-a14a-69b7dde1ecde" containerName="payload-pre-processing" Nov 01 04:07:50.431693 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.431678 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8939b2b-f043-4768-a14a-69b7dde1ecde" containerName="payload-pre-processing" Nov 01 04:07:50.431958 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.431943 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae096853-319c-473e-962e-4c0a1f94eaf3" containerName="payload-processing" Nov 01 04:07:50.432089 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.432077 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda724a5-56d1-4cf5-9b6a-bf5a08fb609e" containerName="payload-processing" Nov 01 04:07:50.432180 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.432170 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8939b2b-f043-4768-a14a-69b7dde1ecde" containerName="payload-pre-processing" Nov 01 04:07:50.432261 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.432251 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2889b1-7ca4-4b1d-821f-1ccd1f9435d2" containerName="payload-pre-processing" Nov 01 04:07:50.458055 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.453209 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.475439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.475408 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf"] Nov 01 04:07:50.524057 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.523891 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dc980b-dockercfg-rklv4\"" Nov 01 04:07:50.541071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.540089 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dc980b\"" Nov 01 04:07:50.566095 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.565062 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f914c94-2cd3-4576-854b-6b0764e120d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.566095 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.565112 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdbb\" (UniqueName: \"kubernetes.io/projected/1f914c94-2cd3-4576-854b-6b0764e120d6-kube-api-access-lgdbb\") pod \"payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.752155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.714268 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb"] Nov 01 04:07:50.752155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.716894 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f914c94-2cd3-4576-854b-6b0764e120d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.752155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.716935 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdbb\" (UniqueName: \"kubernetes.io/projected/1f914c94-2cd3-4576-854b-6b0764e120d6-kube-api-access-lgdbb\") pod \"payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.752155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.719210 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f914c94-2cd3-4576-854b-6b0764e120d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.813539 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.813501 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb"] Nov 01 04:07:50.813718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.813576 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:50.814701 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.814682 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdbb\" (UniqueName: \"kubernetes.io/projected/1f914c94-2cd3-4576-854b-6b0764e120d6-kube-api-access-lgdbb\") pod \"payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.817282 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.817266 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:50.919347 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.919305 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da01e7-4942-4ced-a858-687b8d77ae18-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:50.919523 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:50.919490 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzg8s\" (UniqueName: \"kubernetes.io/projected/20da01e7-4942-4ced-a858-687b8d77ae18-kube-api-access-vzg8s\") pod \"payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:51.021497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.021420 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da01e7-4942-4ced-a858-687b8d77ae18-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:51.021663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.021547 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzg8s\" (UniqueName: \"kubernetes.io/projected/20da01e7-4942-4ced-a858-687b8d77ae18-kube-api-access-vzg8s\") pod \"payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:51.022103 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.022085 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da01e7-4942-4ced-a858-687b8d77ae18-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:51.125875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.125827 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzg8s\" (UniqueName: \"kubernetes.io/projected/20da01e7-4942-4ced-a858-687b8d77ae18-kube-api-access-vzg8s\") pod \"payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:51.182342 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.182307 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:51.282498 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.282380 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf"] Nov 01 04:07:51.670586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.670548 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb"] Nov 01 04:07:51.672087 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:07:51.672056 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20da01e7_4942_4ced_a858_687b8d77ae18.slice/crio-3d03c273368cd76ca9b3082f7615217b6424c9f7e08de0823378a1484f52764c WatchSource:0}: Error finding container 3d03c273368cd76ca9b3082f7615217b6424c9f7e08de0823378a1484f52764c: Status 404 returned error can't find the container with id 3d03c273368cd76ca9b3082f7615217b6424c9f7e08de0823378a1484f52764c Nov 01 04:07:51.948579 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.948487 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" event={"ID":"1f914c94-2cd3-4576-854b-6b0764e120d6","Type":"ContainerStarted","Data":"8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362"} Nov 01 04:07:51.948579 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.948556 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:07:51.948781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.948579 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" event={"ID":"1f914c94-2cd3-4576-854b-6b0764e120d6","Type":"ContainerStarted","Data":"2a6c43473f23bbb949d721578552bf3ee115b14423359dfad9e350dd15055aba"} Nov 01 04:07:51.950255 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.950229 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" event={"ID":"20da01e7-4942-4ced-a858-687b8d77ae18","Type":"ContainerStarted","Data":"dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de"} Nov 01 04:07:51.950411 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.950279 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" event={"ID":"20da01e7-4942-4ced-a858-687b8d77ae18","Type":"ContainerStarted","Data":"3d03c273368cd76ca9b3082f7615217b6424c9f7e08de0823378a1484f52764c"} Nov 01 04:07:51.950411 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:51.950310 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:07:52.071061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:52.068766 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" podStartSLOduration=2.068739238 podStartE2EDuration="2.068739238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:07:52.059100416 +0000 UTC m=+1483.648195045" watchObservedRunningTime="2026-11-01 04:07:52.068739238 +0000 UTC m=+1483.657833868" Nov 01 04:07:52.170491 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:07:52.170337 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" podStartSLOduration=2.170318933 podStartE2EDuration="2.170318933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:07:52.168246129 +0000 UTC m=+1483.757340760" watchObservedRunningTime="2026-11-01 04:07:52.170318933 +0000 UTC m=+1483.759413563" Nov 01 04:08:02.960290 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:02.960249 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:08:02.960767 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:02.960438 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:08:03.002242 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:03.002211 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:08:09.671787 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:09.671694 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:08:09.681848 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:09.681823 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:08:11.145434 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.145391 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj"] Nov 01 04:08:11.151543 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.151521 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.206913 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.206881 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-token\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207161 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-credential-socket\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207218 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae07d12-60f6-44f9-a254-564efe3f15b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207274 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjms\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-kube-api-access-tvjms\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207446 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207408 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-socket\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207446 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207432 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-certs\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207558 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207446 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-data\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207558 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207522 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-envoy\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.207667 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.207620 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-podinfo\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.219742 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.219715 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj"] Nov 01 04:08:11.248565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.248529 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fb1c1f-openshift-default-dockercfg-vdjgl\"" Nov 01 04:08:11.309075 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.309021 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-podinfo\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.310172 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.310142 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-token\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.310824 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.310802 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-credential-socket\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.311540 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.311517 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae07d12-60f6-44f9-a254-564efe3f15b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.313133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.313096 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjms\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-kube-api-access-tvjms\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.313711 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.313689 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-socket\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.314343 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.314325 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-certs\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.314818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.314802 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-data\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.315601 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.315582 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-envoy\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.316196 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.312871 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae07d12-60f6-44f9-a254-564efe3f15b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.316293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.314755 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-certs\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.316293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.311453 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-credential-socket\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.316293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.315299 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-data\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.316293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.314221 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-socket\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.326022 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.325994 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-podinfo\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.342748 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.331373 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-envoy\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.371665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.371599 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-token\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.400291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.400215 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjms\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-kube-api-access-tvjms\") pod \"e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:11.475218 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:11.470373 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:12.147562 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:12.147469 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj"] Nov 01 04:08:12.179059 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:08:12.179009 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae07d12_60f6_44f9_a254_564efe3f15b5.slice/crio-216e34550950b7f497862a1b194b3bb1954577f9fc3c02f1ddf3768f2b7432d2 WatchSource:0}: Error finding container 216e34550950b7f497862a1b194b3bb1954577f9fc3c02f1ddf3768f2b7432d2: Status 404 returned error can't find the container with id 216e34550950b7f497862a1b194b3bb1954577f9fc3c02f1ddf3768f2b7432d2 Nov 01 04:08:12.181749 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:12.181713 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:08:12.181823 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:12.181787 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:08:12.181823 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:12.181816 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:08:13.139081 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:13.139048 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" event={"ID":"9ae07d12-60f6-44f9-a254-564efe3f15b5","Type":"ContainerStarted","Data":"71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e"} Nov 01 04:08:13.139272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:13.139099 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" event={"ID":"9ae07d12-60f6-44f9-a254-564efe3f15b5","Type":"ContainerStarted","Data":"216e34550950b7f497862a1b194b3bb1954577f9fc3c02f1ddf3768f2b7432d2"} Nov 01 04:08:13.313297 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:13.313234 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podStartSLOduration=2.313215691 podStartE2EDuration="2.313215691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:08:13.309488913 +0000 UTC m=+1504.898583542" watchObservedRunningTime="2026-11-01 04:08:13.313215691 +0000 UTC m=+1504.902310321" Nov 01 04:08:13.489059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:13.473197 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:13.514271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:13.514227 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:14.145633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:14.145607 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:14.146597 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:14.146575 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:08:24.610344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:24.610305 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw"] Nov 01 04:08:24.610864 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:24.610563 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" podUID="85375448-d138-4843-bf33-fe52eb296595" containerName="maas-api" containerID="cri-o://ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956" gracePeriod=30 Nov 01 04:08:25.002293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.002267 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:08:25.130602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.130562 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85375448-d138-4843-bf33-fe52eb296595-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85375448-d138-4843-bf33-fe52eb296595-openshift-service-ca\") pod \"85375448-d138-4843-bf33-fe52eb296595\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " Nov 01 04:08:25.139060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.130943 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-metrics-certs\") pod \"85375448-d138-4843-bf33-fe52eb296595\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " Nov 01 04:08:25.139060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.131123 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85375448-d138-4843-bf33-fe52eb296595-kube-api-access-mgshm\" (UniqueName: \"kubernetes.io/projected/85375448-d138-4843-bf33-fe52eb296595-kube-api-access-mgshm\") pod \"85375448-d138-4843-bf33-fe52eb296595\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " Nov 01 04:08:25.139060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.131165 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls\") pod \"85375448-d138-4843-bf33-fe52eb296595\" (UID: \"85375448-d138-4843-bf33-fe52eb296595\") " Nov 01 04:08:25.139060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.135673 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85375448-d138-4843-bf33-fe52eb296595-openshift-service-ca" pod "85375448-d138-4843-bf33-fe52eb296595" (UID: "85375448-d138-4843-bf33-fe52eb296595"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:08:25.167092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.166933 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls" pod "85375448-d138-4843-bf33-fe52eb296595" (UID: "85375448-d138-4843-bf33-fe52eb296595"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:08:25.184252 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.184209 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85375448-d138-4843-bf33-fe52eb296595-kube-api-access-mgshm" pod "85375448-d138-4843-bf33-fe52eb296595" (UID: "85375448-d138-4843-bf33-fe52eb296595"). InnerVolumeSpecName "kube-api-access-mgshm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:25.198571 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.198508 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-metrics-certs" pod "85375448-d138-4843-bf33-fe52eb296595" (UID: "85375448-d138-4843-bf33-fe52eb296595"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:08:25.266489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.260392 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgshm\" (UniqueName: \"kubernetes.io/projected/85375448-d138-4843-bf33-fe52eb296595-kube-api-access-mgshm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:25.266489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.266202 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:25.266489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.266236 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85375448-d138-4843-bf33-fe52eb296595-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:25.266489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.266249 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85375448-d138-4843-bf33-fe52eb296595-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:25.299419 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.299385 6735 generic.go:356] "Generic (PLEG): container finished" podID="85375448-d138-4843-bf33-fe52eb296595" containerID="ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956" exitCode=0 Nov 01 04:08:25.299602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.299459 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" event={"ID":"85375448-d138-4843-bf33-fe52eb296595","Type":"ContainerDied","Data":"ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956"} Nov 01 04:08:25.299602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.299494 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" Nov 01 04:08:25.299602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.299507 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw" event={"ID":"85375448-d138-4843-bf33-fe52eb296595","Type":"ContainerDied","Data":"fa92180122c754ec4b9b747399ddb4f1f2e0300104dbf333cd03c09ec5372cb0"} Nov 01 04:08:25.299602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.299539 6735 scope.go:122] "RemoveContainer" containerID="ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956" Nov 01 04:08:25.327197 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.327179 6735 scope.go:122] "RemoveContainer" containerID="ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956" Nov 01 04:08:25.327563 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:25.327527 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956\": container with ID starting with ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956 not found: ID does not exist" containerID="ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956" Nov 01 04:08:25.327652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.327570 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956"} err="failed to get container status \"ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956\": rpc error: code = NotFound desc = could not find container \"ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956\": container with ID starting with ccbb0f63d7c37eb718509d940312a7caf48c9154d4e722967b2f565fb3dfe956 not found: ID does not exist" Nov 01 04:08:25.455828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.455794 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw"] Nov 01 04:08:25.459661 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:25.459638 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-182e51-698b867859-6v4nw"] Nov 01 04:08:26.299585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:26.299552 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:08:26.558006 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:26.557909 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85375448-d138-4843-bf33-fe52eb296595" path="/var/lib/kubelet/pods/85375448-d138-4843-bf33-fe52eb296595/volumes" Nov 01 04:08:32.572495 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:32.572457 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv"] Nov 01 04:08:32.572908 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:32.572715 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" containerID="cri-o://7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0" gracePeriod=30 Nov 01 04:08:32.620789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:32.620719 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64"] Nov 01 04:08:32.621130 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:32.620969 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" containerID="cri-o://f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7" gracePeriod=30 Nov 01 04:08:33.002624 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:33.002590 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:08:34.992061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:34.986972 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf"] Nov 01 04:08:34.992061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:34.988108 6735 cpu_manager.go:414] "containerMap: removing container" podUID="85375448-d138-4843-bf33-fe52eb296595" containerName="maas-api" Nov 01 04:08:34.992061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:34.988130 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85375448-d138-4843-bf33-fe52eb296595" containerName="maas-api" Nov 01 04:08:34.992061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:34.988387 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="85375448-d138-4843-bf33-fe52eb296595" containerName="maas-api" Nov 01 04:08:35.006185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.006154 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.042147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.036799 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08d53b8-9e19-45b5-8059-8b69fbc21927-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.042147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.037152 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gvlg\" (UniqueName: \"kubernetes.io/projected/c08d53b8-9e19-45b5-8059-8b69fbc21927-kube-api-access-7gvlg\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.100149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.081238 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf"] Nov 01 04:08:35.100149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.082255 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-078e0d65\"" Nov 01 04:08:35.100149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.082450 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-078e0d65-dockercfg-srxxx\"" Nov 01 04:08:35.201061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.145058 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gvlg\" (UniqueName: \"kubernetes.io/projected/c08d53b8-9e19-45b5-8059-8b69fbc21927-kube-api-access-7gvlg\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.201061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.145181 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08d53b8-9e19-45b5-8059-8b69fbc21927-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.201061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.146361 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08d53b8-9e19-45b5-8059-8b69fbc21927-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.387707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.387238 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz"] Nov 01 04:08:35.387707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.387567 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gvlg\" (UniqueName: \"kubernetes.io/projected/c08d53b8-9e19-45b5-8059-8b69fbc21927-kube-api-access-7gvlg\") pod \"payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.398271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.398240 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz"] Nov 01 04:08:35.398438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.398358 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.402926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.402889 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024635e2-f5c8-465f-8833-4488e8da9d14-plugins-config-volume\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.403182 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.403080 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpdww\" (UniqueName: \"kubernetes.io/projected/024635e2-f5c8-465f-8833-4488e8da9d14-kube-api-access-jpdww\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.504657 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.504624 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpdww\" (UniqueName: \"kubernetes.io/projected/024635e2-f5c8-465f-8833-4488e8da9d14-kube-api-access-jpdww\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.504919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.504894 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024635e2-f5c8-465f-8833-4488e8da9d14-plugins-config-volume\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.506130 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.505715 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024635e2-f5c8-465f-8833-4488e8da9d14-plugins-config-volume\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.557514 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.557484 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:35.562853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.562822 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpdww\" (UniqueName: \"kubernetes.io/projected/024635e2-f5c8-465f-8833-4488e8da9d14-kube-api-access-jpdww\") pod \"payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.723390 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.723360 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:35.931165 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:35.931136 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf"] Nov 01 04:08:35.932713 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:08:35.932684 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08d53b8_9e19_45b5_8059_8b69fbc21927.slice/crio-95209985509c38c4dec7033c93b1062996f158a109dc0619b9986a00ee574d96 WatchSource:0}: Error finding container 95209985509c38c4dec7033c93b1062996f158a109dc0619b9986a00ee574d96: Status 404 returned error can't find the container with id 95209985509c38c4dec7033c93b1062996f158a109dc0619b9986a00ee574d96 Nov 01 04:08:36.096298 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.096266 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz"] Nov 01 04:08:36.097782 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:08:36.097753 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024635e2_f5c8_465f_8833_4488e8da9d14.slice/crio-327d6d7528ee9e7a3542820dca68fc9ec716ff89e6a04e6aa0009f9eb88b27af WatchSource:0}: Error finding container 327d6d7528ee9e7a3542820dca68fc9ec716ff89e6a04e6aa0009f9eb88b27af: Status 404 returned error can't find the container with id 327d6d7528ee9e7a3542820dca68fc9ec716ff89e6a04e6aa0009f9eb88b27af Nov 01 04:08:36.373758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.373661 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4"] Nov 01 04:08:36.373915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.373849 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" containerID="cri-o://b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b" gracePeriod=30 Nov 01 04:08:36.409376 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.409339 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" event={"ID":"c08d53b8-9e19-45b5-8059-8b69fbc21927","Type":"ContainerStarted","Data":"db2a5682f05726a361458e87ad11c8301d3801c12f7c097191707e9781ff9873"} Nov 01 04:08:36.409638 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.409606 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" event={"ID":"c08d53b8-9e19-45b5-8059-8b69fbc21927","Type":"ContainerStarted","Data":"95209985509c38c4dec7033c93b1062996f158a109dc0619b9986a00ee574d96"} Nov 01 04:08:36.409733 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.409644 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:36.411395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.411372 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" event={"ID":"024635e2-f5c8-465f-8833-4488e8da9d14","Type":"ContainerStarted","Data":"a837a84b1f43b0ed7e3a896172f40fe6f93db5cc2ecd1b0239b9e9738b30dd72"} Nov 01 04:08:36.411506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.411413 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" event={"ID":"024635e2-f5c8-465f-8833-4488e8da9d14","Type":"ContainerStarted","Data":"327d6d7528ee9e7a3542820dca68fc9ec716ff89e6a04e6aa0009f9eb88b27af"} Nov 01 04:08:36.411506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.411467 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:36.605350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.605300 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" podStartSLOduration=2.605286899 podStartE2EDuration="2.605286899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:08:36.602427785 +0000 UTC m=+1528.191522416" watchObservedRunningTime="2026-11-01 04:08:36.605286899 +0000 UTC m=+1528.194381528" Nov 01 04:08:36.714183 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:36.714117 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" podStartSLOduration=1.714103012 podStartE2EDuration="1.714103012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:08:36.711237499 +0000 UTC m=+1528.300332129" watchObservedRunningTime="2026-11-01 04:08:36.714103012 +0000 UTC m=+1528.303197644" Nov 01 04:08:37.918329 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.918302 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:08:37.942476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.942449 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:08:37.946616 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946598 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b931116-382f-4f97-83a1-90befaf22f2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931116-382f-4f97-83a1-90befaf22f2e-istiod-ca-cert\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.946673 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946633 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-socket\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.946722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946683 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-credential-socket\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.946722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946714 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b931116-382f-4f97-83a1-90befaf22f2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931116-382f-4f97-83a1-90befaf22f2e-istio-podinfo\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.946818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946750 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-istio-token\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.946940 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946924 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-certs\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.946997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946964 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-envoy\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.946997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946972 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-socket" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:37.947126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.946990 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-credential-socket" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:37.947126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947020 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b931116-382f-4f97-83a1-90befaf22f2e-istiod-ca-cert" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:08:37.947126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947024 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-data\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.947126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947096 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-kube-api-access-jwbm2\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-kube-api-access-jwbm2\") pod \"3b931116-382f-4f97-83a1-90befaf22f2e\" (UID: \"3b931116-382f-4f97-83a1-90befaf22f2e\") " Nov 01 04:08:37.947336 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947246 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-certs" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:37.947336 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947260 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-data" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:37.947762 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947745 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b931116-382f-4f97-83a1-90befaf22f2e-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:37.947813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947766 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:37.947813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947775 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:37.947813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947786 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:37.947813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.947799 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:37.949823 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.949799 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b931116-382f-4f97-83a1-90befaf22f2e-istio-podinfo" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:08:37.949901 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.949830 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-kube-api-access-jwbm2" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "kube-api-access-jwbm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:37.949901 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.949844 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-envoy" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:37.949901 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:37.949861 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-istio-token" pod "3b931116-382f-4f97-83a1-90befaf22f2e" (UID: "3b931116-382f-4f97-83a1-90befaf22f2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:38.048840 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.048808 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-kube-api-access-bk5tn\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-kube-api-access-bk5tn\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.048871 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-certs\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.048920 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-data\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.048960 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-podinfo\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049050 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-token\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-token\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049219 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-credential-socket\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049236 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-data" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:38.049325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049261 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-envoy\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049519 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049257 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-certs" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:38.049519 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049339 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-socket\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049519 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049380 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istiod-ca-cert\") pod \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\" (UID: \"9f336ac6-ee8b-4bc8-963f-e1da031cb351\") " Nov 01 04:08:38.049519 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049458 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-credential-socket" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:38.049742 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.049590 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-socket" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:38.050023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050003 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.050126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050054 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.050126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050069 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b931116-382f-4f97-83a1-90befaf22f2e-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.050126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050082 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.050126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050096 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwbm2\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-kube-api-access-jwbm2\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.050126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050109 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.050126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050007 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istiod-ca-cert" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:08:38.050331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050122 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b931116-382f-4f97-83a1-90befaf22f2e-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.050331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.050170 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b931116-382f-4f97-83a1-90befaf22f2e-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.051510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.051489 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-envoy" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:38.051568 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.051521 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-token" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:38.051828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.051807 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-podinfo" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:08:38.051828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.051813 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-kube-api-access-bk5tn" pod "9f336ac6-ee8b-4bc8-963f-e1da031cb351" (UID: "9f336ac6-ee8b-4bc8-963f-e1da031cb351"). InnerVolumeSpecName "kube-api-access-bk5tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:38.152004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.151924 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.152004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.151951 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5tn\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-kube-api-access-bk5tn\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.152004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.151962 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.152004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.151971 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.152004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.151980 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f336ac6-ee8b-4bc8-963f-e1da031cb351-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:38.425348 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.425264 6735 generic.go:356] "Generic (PLEG): container finished" podID="3b931116-382f-4f97-83a1-90befaf22f2e" containerID="7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0" exitCode=0 Nov 01 04:08:38.425507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.425350 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" Nov 01 04:08:38.425507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.425355 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" event={"ID":"3b931116-382f-4f97-83a1-90befaf22f2e","Type":"ContainerDied","Data":"7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0"} Nov 01 04:08:38.425507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.425414 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" event={"ID":"3b931116-382f-4f97-83a1-90befaf22f2e","Type":"ContainerDied","Data":"5e21e905fc610bb52b2bff00c28cbc036313ec79c1da0b67573804acff4a0cce"} Nov 01 04:08:38.425507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.425442 6735 scope.go:122] "RemoveContainer" containerID="7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0" Nov 01 04:08:38.427176 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.427147 6735 generic.go:356] "Generic (PLEG): container finished" podID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerID="f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7" exitCode=0 Nov 01 04:08:38.427306 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.427180 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" event={"ID":"9f336ac6-ee8b-4bc8-963f-e1da031cb351","Type":"ContainerDied","Data":"f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7"} Nov 01 04:08:38.427306 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.427189 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" Nov 01 04:08:38.427306 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.427218 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64" event={"ID":"9f336ac6-ee8b-4bc8-963f-e1da031cb351","Type":"ContainerDied","Data":"2ccb9f9f107fe14fdaa021ac331b3367ca1145bd8bb83b7d9220f5ebfc851c6f"} Nov 01 04:08:38.437926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.437907 6735 scope.go:122] "RemoveContainer" containerID="7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0" Nov 01 04:08:38.438230 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:38.438211 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0\": container with ID starting with 7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0 not found: ID does not exist" containerID="7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0" Nov 01 04:08:38.438315 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.438241 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0"} err="failed to get container status \"7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0\": rpc error: code = NotFound desc = could not find container \"7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0\": container with ID starting with 7fb9194f4c5da2d233105fcaf7bb0358e1c07b5d01eefef461a8a5531e0641a0 not found: ID does not exist" Nov 01 04:08:38.438315 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.438266 6735 scope.go:122] "RemoveContainer" containerID="f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7" Nov 01 04:08:38.447875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.447842 6735 scope.go:122] "RemoveContainer" containerID="f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7" Nov 01 04:08:38.448218 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:38.448198 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7\": container with ID starting with f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7 not found: ID does not exist" containerID="f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7" Nov 01 04:08:38.448310 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.448225 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7"} err="failed to get container status \"f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7\": rpc error: code = NotFound desc = could not find container \"f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7\": container with ID starting with f03f36f89ee171aef11a33e50c5c6dfbd361e688fba55e140f282561bab522d7 not found: ID does not exist" Nov 01 04:08:38.580994 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.580960 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64"] Nov 01 04:08:38.626907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.626875 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-l2v64"] Nov 01 04:08:38.688354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.688271 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv"] Nov 01 04:08:38.698830 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.698793 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv"] Nov 01 04:08:38.897684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.897646 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 04:08:38.897872 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:38.897703 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-vnhhv" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 04:08:40.526972 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:40.526936 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" path="/var/lib/kubelet/pods/3b931116-382f-4f97-83a1-90befaf22f2e/volumes" Nov 01 04:08:40.527670 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:40.527649 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" path="/var/lib/kubelet/pods/9f336ac6-ee8b-4bc8-963f-e1da031cb351/volumes" Nov 01 04:08:41.664637 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.664608 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:08:41.707454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707421 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istiod-ca-cert\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.707663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707623 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-socket\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.707730 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707703 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-data\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.707792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707737 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-token\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-token\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.707792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707770 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istiod-ca-cert" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:08:41.707896 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707783 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-envoy\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.707951 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707900 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-socket" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:41.708007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707964 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-certs\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.708007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.707991 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-data" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:41.708137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708016 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-kube-api-access-7czmb\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-kube-api-access-7czmb\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.708137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708092 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-credential-socket\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.708137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708116 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-podinfo\") pod \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\" (UID: \"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d\") " Nov 01 04:08:41.708289 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708128 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-certs" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:41.708351 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708303 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-credential-socket" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:41.708689 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708664 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.708838 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708822 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.708906 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708844 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.708906 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708859 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.708906 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.708872 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.710409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.710380 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-token" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:41.710522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.710437 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-kube-api-access-7czmb" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "kube-api-access-7czmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:41.710699 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.710678 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-podinfo" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:08:41.710797 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.710683 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-envoy" pod "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" (UID: "6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:41.809707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.809678 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.809707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.809705 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.809899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.809716 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czmb\" (UniqueName: \"kubernetes.io/projected/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-kube-api-access-7czmb\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:41.809899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:41.809725 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:42.351115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351083 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl"] Nov 01 04:08:42.351786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351772 6735 cpu_manager.go:414] "containerMap: removing container" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" Nov 01 04:08:42.351844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351788 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" Nov 01 04:08:42.351844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351802 6735 cpu_manager.go:414] "containerMap: removing container" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" Nov 01 04:08:42.351844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351807 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" Nov 01 04:08:42.351937 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351850 6735 cpu_manager.go:414] "containerMap: removing container" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" Nov 01 04:08:42.351937 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351855 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" Nov 01 04:08:42.352003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351945 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b931116-382f-4f97-83a1-90befaf22f2e" containerName="istio-proxy" Nov 01 04:08:42.352003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351956 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerName="istio-proxy" Nov 01 04:08:42.352003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.351964 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f336ac6-ee8b-4bc8-963f-e1da031cb351" containerName="istio-proxy" Nov 01 04:08:42.356837 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.356814 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.408025 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.407994 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dc980b-serving-cert\"" Nov 01 04:08:42.412430 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.412403 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl"] Nov 01 04:08:42.417024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.417001 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqjw\" (UniqueName: \"kubernetes.io/projected/62a4a366-e024-4bb5-87f0-848a646c136a-kube-api-access-zlqjw\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.417176 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.417159 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-metrics-certs\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.417234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.417199 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.417411 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.417389 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4a366-e024-4bb5-87f0-848a646c136a-openshift-service-ca\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.458224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.458187 6735 generic.go:356] "Generic (PLEG): container finished" podID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" containerID="b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b" exitCode=0 Nov 01 04:08:42.458361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.458237 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" Nov 01 04:08:42.458361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.458267 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" event={"ID":"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d","Type":"ContainerDied","Data":"b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b"} Nov 01 04:08:42.458361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.458320 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4" event={"ID":"6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d","Type":"ContainerDied","Data":"26c930e97ff8fcd1cda19a6e7e215c73192950cc51f04f1bdc4a72e667e8b507"} Nov 01 04:08:42.458361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.458342 6735 scope.go:122] "RemoveContainer" containerID="b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b" Nov 01 04:08:42.469150 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.469126 6735 scope.go:122] "RemoveContainer" containerID="b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b" Nov 01 04:08:42.469460 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:42.469439 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b\": container with ID starting with b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b not found: ID does not exist" containerID="b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b" Nov 01 04:08:42.469526 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.469469 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b"} err="failed to get container status \"b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b\": rpc error: code = NotFound desc = could not find container \"b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b\": container with ID starting with b7dd7a57c27c29f8985e2bf5b0daf4c6f2fe0bc8c40231b5f1e52494be193c7b not found: ID does not exist" Nov 01 04:08:42.520054 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.518585 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqjw\" (UniqueName: \"kubernetes.io/projected/62a4a366-e024-4bb5-87f0-848a646c136a-kube-api-access-zlqjw\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.520054 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.518702 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-metrics-certs\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.520054 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.518751 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.520054 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.518911 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4a366-e024-4bb5-87f0-848a646c136a-openshift-service-ca\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.520054 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:42.518944 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-serving-cert: secret "maas-api-e2e-api-b-dc980b-serving-cert" not found Nov 01 04:08:42.520054 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:42.519060 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls podName:62a4a366-e024-4bb5-87f0-848a646c136a nodeName:}" failed. No retries permitted until 2026-11-01 04:08:43.019015162 +0000 UTC m=+1534.608109786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls") pod "maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" (UID: "62a4a366-e024-4bb5-87f0-848a646c136a") : secret "maas-api-e2e-api-b-dc980b-serving-cert" not found Nov 01 04:08:42.520054 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.519623 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4a366-e024-4bb5-87f0-848a646c136a-openshift-service-ca\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.522584 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.522555 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-metrics-certs\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.575752 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.575725 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4"] Nov 01 04:08:42.579588 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.579566 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqjw\" (UniqueName: \"kubernetes.io/projected/62a4a366-e024-4bb5-87f0-848a646c136a-kube-api-access-zlqjw\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:42.627105 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:42.625015 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-z7jw4"] Nov 01 04:08:43.028478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:43.028446 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:43.031078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:43.031024 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls\") pod \"maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:43.272340 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:43.272312 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:43.649541 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:43.649511 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl"] Nov 01 04:08:43.650907 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:08:43.650882 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a4a366_e024_4bb5_87f0_848a646c136a.slice/crio-118502a47c2de646e582e4728edb8159c6085444dc4f5eef1c2d187b2e214938 WatchSource:0}: Error finding container 118502a47c2de646e582e4728edb8159c6085444dc4f5eef1c2d187b2e214938: Status 404 returned error can't find the container with id 118502a47c2de646e582e4728edb8159c6085444dc4f5eef1c2d187b2e214938 Nov 01 04:08:44.473496 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:44.473465 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" event={"ID":"62a4a366-e024-4bb5-87f0-848a646c136a","Type":"ContainerStarted","Data":"118502a47c2de646e582e4728edb8159c6085444dc4f5eef1c2d187b2e214938"} Nov 01 04:08:44.571203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:44.571171 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d" path="/var/lib/kubelet/pods/6cdcf4ef-8793-44c3-8b13-a6e76e0cdb9d/volumes" Nov 01 04:08:45.524059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:45.523273 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" event={"ID":"62a4a366-e024-4bb5-87f0-848a646c136a","Type":"ContainerStarted","Data":"9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0"} Nov 01 04:08:45.525266 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:45.525016 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:45.640222 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:45.640155 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" podStartSLOduration=2.582750375 podStartE2EDuration="3.640140751s" totalImagesPullingTime="1.057390376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:08:45.637600491 +0000 UTC m=+1537.226695154" watchObservedRunningTime="2026-11-01 04:08:45.640140751 +0000 UTC m=+1537.229235408" Nov 01 04:08:47.421136 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:47.421098 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:08:47.421626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:47.421220 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:08:50.956944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:50.956904 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl"] Nov 01 04:08:50.957523 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:50.957149 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" containerID="cri-o://c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5" gracePeriod=30 Nov 01 04:08:51.003553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:51.003520 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97"] Nov 01 04:08:51.003776 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:51.003753 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" containerID="cri-o://f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882" gracePeriod=30 Nov 01 04:08:52.563778 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:52.563746 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:08:56.302735 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.302705 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:08:56.328720 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.328696 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:08:56.400599 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400561 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-token\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-token\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.400763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400626 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-kube-api-access-cpchh\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-kube-api-access-cpchh\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.400763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400688 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-certs\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.400763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400710 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-podinfo\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.400763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400732 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-credential-socket\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.400992 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400794 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-certs\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.400992 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400822 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-token\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-token\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.400992 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400885 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-data\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.400992 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.400951 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-envoy\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.401234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401011 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-podinfo\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.401234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401016 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-credential-socket" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.401234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401068 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-certs" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.401234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401129 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-credential-socket\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.401234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401181 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-socket\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.401234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401220 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-envoy\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.401586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401271 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-socket\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.401586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401305 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-data\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.401586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401333 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c79d751-1e54-43b5-8068-61207d04bfcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79d751-1e54-43b5-8068-61207d04bfcc-istiod-ca-cert\") pod \"6c79d751-1e54-43b5-8068-61207d04bfcc\" (UID: \"6c79d751-1e54-43b5-8068-61207d04bfcc\") " Nov 01 04:08:56.401586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401343 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-data" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.401586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401381 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istiod-ca-cert\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.401586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401449 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-kube-api-access-hk7bm\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-kube-api-access-hk7bm\") pod \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\" (UID: \"74bc5c87-ce74-4d5b-9f18-37a314a6a483\") " Nov 01 04:08:56.401586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401488 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-certs" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.401921 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.401662 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-socket" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.402078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402026 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-data" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.402298 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402278 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-credential-socket" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402445 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402474 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402490 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402505 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402519 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402534 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402645 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c79d751-1e54-43b5-8068-61207d04bfcc-istiod-ca-cert" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:08:56.402763 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.402751 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-socket" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.403261 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.403093 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istiod-ca-cert" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:08:56.404118 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.404077 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-podinfo" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:08:56.404702 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.404591 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-podinfo" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:08:56.405635 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.405552 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-token" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:56.405635 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.405600 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-kube-api-access-hk7bm" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "kube-api-access-hk7bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:56.406287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.406264 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-kube-api-access-cpchh" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "kube-api-access-cpchh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:56.406442 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.406379 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-token" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:08:56.406582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.406558 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-envoy" pod "74bc5c87-ce74-4d5b-9f18-37a314a6a483" (UID: "74bc5c87-ce74-4d5b-9f18-37a314a6a483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.412474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.412452 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-envoy" pod "6c79d751-1e54-43b5-8068-61207d04bfcc" (UID: "6c79d751-1e54-43b5-8068-61207d04bfcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:08:56.503938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503904 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.503938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503933 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.503938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503942 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503951 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503961 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74bc5c87-ce74-4d5b-9f18-37a314a6a483-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503970 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503981 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79d751-1e54-43b5-8068-61207d04bfcc-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503989 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.503998 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7bm\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-kube-api-access-hk7bm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.504009 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74bc5c87-ce74-4d5b-9f18-37a314a6a483-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.504019 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpchh\" (UniqueName: \"kubernetes.io/projected/6c79d751-1e54-43b5-8068-61207d04bfcc-kube-api-access-cpchh\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.504198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.504027 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79d751-1e54-43b5-8068-61207d04bfcc-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:08:56.606853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.606820 6735 generic.go:356] "Generic (PLEG): container finished" podID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerID="c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5" exitCode=0 Nov 01 04:08:56.607055 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.606922 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" event={"ID":"6c79d751-1e54-43b5-8068-61207d04bfcc","Type":"ContainerDied","Data":"c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5"} Nov 01 04:08:56.607055 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.606973 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" event={"ID":"6c79d751-1e54-43b5-8068-61207d04bfcc","Type":"ContainerDied","Data":"df32a8fa3e5ead1ad0816f26742283878879a463d9dc4f91c219d0be150847e5"} Nov 01 04:08:56.607055 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.607002 6735 scope.go:122] "RemoveContainer" containerID="c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5" Nov 01 04:08:56.607248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.607206 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl" Nov 01 04:08:56.610893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.609647 6735 generic.go:356] "Generic (PLEG): container finished" podID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerID="f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882" exitCode=0 Nov 01 04:08:56.610893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.609795 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" event={"ID":"74bc5c87-ce74-4d5b-9f18-37a314a6a483","Type":"ContainerDied","Data":"f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882"} Nov 01 04:08:56.610893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.609840 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" event={"ID":"74bc5c87-ce74-4d5b-9f18-37a314a6a483","Type":"ContainerDied","Data":"0dc95c7d26b8c1a0f24ef8af9b4cca1865b428bf980bf383d276e29b82fe9609"} Nov 01 04:08:56.610893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.609912 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97" Nov 01 04:08:56.618966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.618943 6735 scope.go:122] "RemoveContainer" containerID="c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5" Nov 01 04:08:56.619362 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:56.619332 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5\": container with ID starting with c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5 not found: ID does not exist" containerID="c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5" Nov 01 04:08:56.619429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.619373 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5"} err="failed to get container status \"c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5\": rpc error: code = NotFound desc = could not find container \"c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5\": container with ID starting with c1b1a8da5ed567c415daaf63f97f5c5e92ffdbbabed02f9d4a4715bbb2009ae5 not found: ID does not exist" Nov 01 04:08:56.619429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.619397 6735 scope.go:122] "RemoveContainer" containerID="f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882" Nov 01 04:08:56.631704 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.631685 6735 scope.go:122] "RemoveContainer" containerID="f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882" Nov 01 04:08:56.635305 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:08:56.635271 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882\": container with ID starting with f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882 not found: ID does not exist" containerID="f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882" Nov 01 04:08:56.635402 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.635314 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882"} err="failed to get container status \"f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882\": rpc error: code = NotFound desc = could not find container \"f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882\": container with ID starting with f4cb3d082a083f42ccd39c84760e9d42f0ada80db827923ad42e30a84706d882 not found: ID does not exist" Nov 01 04:08:56.762664 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.762621 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl"] Nov 01 04:08:56.768516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.768477 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-xt8wl"] Nov 01 04:08:56.871000 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.870904 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97"] Nov 01 04:08:56.873601 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:56.873582 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-gkd97"] Nov 01 04:08:58.529379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:58.529337 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" path="/var/lib/kubelet/pods/6c79d751-1e54-43b5-8068-61207d04bfcc/volumes" Nov 01 04:08:58.529876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:58.529861 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" path="/var/lib/kubelet/pods/74bc5c87-ce74-4d5b-9f18-37a314a6a483/volumes" Nov 01 04:08:58.876381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:08:58.876300 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:09:03.002306 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:03.002266 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:09:22.368466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.368424 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt"] Nov 01 04:09:22.369234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.369213 6735 cpu_manager.go:414] "containerMap: removing container" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" Nov 01 04:09:22.369234 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.369235 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" Nov 01 04:09:22.369394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.369247 6735 cpu_manager.go:414] "containerMap: removing container" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" Nov 01 04:09:22.369394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.369252 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" Nov 01 04:09:22.369394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.369371 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bc5c87-ce74-4d5b-9f18-37a314a6a483" containerName="istio-proxy" Nov 01 04:09:22.369394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.369379 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c79d751-1e54-43b5-8068-61207d04bfcc" containerName="istio-proxy" Nov 01 04:09:22.374611 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.374592 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.471794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.471757 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-078e0d65-serving-cert\"" Nov 01 04:09:22.474653 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.474634 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt"] Nov 01 04:09:22.499715 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.499675 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbvn\" (UniqueName: \"kubernetes.io/projected/7f000817-dda8-47bf-961b-bab5deefd69a-kube-api-access-hlbvn\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.499836 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.499796 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f000817-dda8-47bf-961b-bab5deefd69a-openshift-service-ca\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.499890 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.499877 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-metrics-certs\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.499997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.499977 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.600849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.600819 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbvn\" (UniqueName: \"kubernetes.io/projected/7f000817-dda8-47bf-961b-bab5deefd69a-kube-api-access-hlbvn\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.601088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.600907 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f000817-dda8-47bf-961b-bab5deefd69a-openshift-service-ca\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.601088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.600997 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-metrics-certs\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.601088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.601063 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.601249 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:22.601200 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-serving-cert: secret "maas-api-e2e-mt-078e0d65-serving-cert" not found Nov 01 04:09:22.601289 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:22.601271 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls podName:7f000817-dda8-47bf-961b-bab5deefd69a nodeName:}" failed. No retries permitted until 2026-11-01 04:09:23.10125169 +0000 UTC m=+1574.690346300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls") pod "maas-api-e2e-mt-078e0d65-6588697649-j7dmt" (UID: "7f000817-dda8-47bf-961b-bab5deefd69a") : secret "maas-api-e2e-mt-078e0d65-serving-cert" not found Nov 01 04:09:22.601657 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.601638 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f000817-dda8-47bf-961b-bab5deefd69a-openshift-service-ca\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.603478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.603462 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-metrics-certs\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:22.688859 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:22.688779 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbvn\" (UniqueName: \"kubernetes.io/projected/7f000817-dda8-47bf-961b-bab5deefd69a-kube-api-access-hlbvn\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:23.119804 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:23.119768 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:23.122371 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:23.122352 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls\") pod \"maas-api-e2e-mt-078e0d65-6588697649-j7dmt\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:23.288513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:23.288483 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:23.660754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:23.660720 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt"] Nov 01 04:09:23.662142 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:09:23.662116 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f000817_dda8_47bf_961b_bab5deefd69a.slice/crio-ca48199fb818668219a72fc56c3b044db257d9a7ed49a5d284c7ea1db655e3b9 WatchSource:0}: Error finding container ca48199fb818668219a72fc56c3b044db257d9a7ed49a5d284c7ea1db655e3b9: Status 404 returned error can't find the container with id ca48199fb818668219a72fc56c3b044db257d9a7ed49a5d284c7ea1db655e3b9 Nov 01 04:09:23.784702 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:23.784668 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" event={"ID":"7f000817-dda8-47bf-961b-bab5deefd69a","Type":"ContainerStarted","Data":"ca48199fb818668219a72fc56c3b044db257d9a7ed49a5d284c7ea1db655e3b9"} Nov 01 04:09:24.792427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:24.792335 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" event={"ID":"7f000817-dda8-47bf-961b-bab5deefd69a","Type":"ContainerStarted","Data":"14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9"} Nov 01 04:09:24.792801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:24.792442 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:24.905717 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:24.905664 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" podStartSLOduration=2.089494021 podStartE2EDuration="2.90564751s" totalImagesPullingTime="816.153489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:09:24.90346113 +0000 UTC m=+1576.492555772" watchObservedRunningTime="2026-11-01 04:09:24.90564751 +0000 UTC m=+1576.494742144" Nov 01 04:09:30.409239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:30.409200 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:09:30.833252 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:30.833221 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:09:31.941917 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:31.941872 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc"] Nov 01 04:09:31.942421 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:31.942143 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" podUID="10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" containerName="maas-api" containerID="cri-o://d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2" gracePeriod=30 Nov 01 04:09:32.460115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.458296 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:09:32.539636 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.539599 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls\") pod \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " Nov 01 04:09:32.540091 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.540070 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-metrics-certs\") pod \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " Nov 01 04:09:32.540196 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.540184 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-kube-api-access-lnfbv\" (UniqueName: \"kubernetes.io/projected/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-kube-api-access-lnfbv\") pod \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " Nov 01 04:09:32.540271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.540247 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-openshift-service-ca\") pod \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\" (UID: \"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276\") " Nov 01 04:09:32.541655 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.541623 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-openshift-service-ca" pod "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" (UID: "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:09:32.542180 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.542156 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls" pod "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" (UID: "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:09:32.547136 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.545084 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-metrics-certs" pod "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" (UID: "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:09:32.547136 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.545192 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-kube-api-access-lnfbv" pod "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" (UID: "10e5bcb3-80b0-4aa0-a000-0ab53b6cc276"). InnerVolumeSpecName "kube-api-access-lnfbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:09:32.641891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.641856 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfbv\" (UniqueName: \"kubernetes.io/projected/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-kube-api-access-lnfbv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:32.641891 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.641892 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:32.642198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.641907 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:32.642198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.641920 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:32.885327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.885240 6735 generic.go:356] "Generic (PLEG): container finished" podID="10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" containerID="d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2" exitCode=0 Nov 01 04:09:32.885726 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.885702 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" event={"ID":"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276","Type":"ContainerDied","Data":"d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2"} Nov 01 04:09:32.885877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.885858 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" event={"ID":"10e5bcb3-80b0-4aa0-a000-0ab53b6cc276","Type":"ContainerDied","Data":"e6eae3ef4e0506366c556a346c64059f8f5bea667662091da4f64c56f8bac151"} Nov 01 04:09:32.886120 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.886101 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc" Nov 01 04:09:32.893800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.893773 6735 scope.go:122] "RemoveContainer" containerID="d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2" Nov 01 04:09:32.997743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:32.993275 6735 scope.go:122] "RemoveContainer" containerID="d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2" Nov 01 04:09:33.015056 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:33.011619 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2\": container with ID starting with d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2 not found: ID does not exist" containerID="d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2" Nov 01 04:09:33.015056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:33.011664 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2"} err="failed to get container status \"d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2\": rpc error: code = NotFound desc = could not find container \"d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2\": container with ID starting with d03ae25a297a89514b3b9540340a8455729ea85277c42d0e54d4b05dff2d2fd2 not found: ID does not exist" Nov 01 04:09:33.078982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:33.078920 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc"] Nov 01 04:09:33.117302 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:33.117254 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aab51f-5f8c74c6f7-ndwdc"] Nov 01 04:09:33.129262 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:33.129228 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:09:34.527874 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:34.527832 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" path="/var/lib/kubelet/pods/10e5bcb3-80b0-4aa0-a000-0ab53b6cc276/volumes" Nov 01 04:09:40.939622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.939530 6735 generic.go:356] "Generic (PLEG): container finished" podID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerID="71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e" exitCode=0 Nov 01 04:09:40.939622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.939570 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" event={"ID":"9ae07d12-60f6-44f9-a254-564efe3f15b5","Type":"ContainerDied","Data":"71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e"} Nov 01 04:09:40.940105 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.939974 6735 scope.go:122] "RemoveContainer" containerID="71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e" Nov 01 04:09:40.941654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.941623 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:09:40.941760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.941690 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:09:40.941760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.941725 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:09:40.955028 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.955004 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr"] Nov 01 04:09:40.955261 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:40.955240 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" containerID="cri-o://8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f" gracePeriod=30 Nov 01 04:09:41.973073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:41.973005 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" event={"ID":"9ae07d12-60f6-44f9-a254-564efe3f15b5","Type":"ContainerStarted","Data":"6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd"} Nov 01 04:09:42.470933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:42.470905 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:09:42.472078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:42.472020 6735 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 04:09:42.472203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:42.472110 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 04:09:43.471563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:43.471522 6735 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 04:09:43.471976 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:43.471606 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 04:09:44.160062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:44.156148 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:09:44.474487 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:44.474454 6735 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 04:09:44.475191 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:44.475159 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 04:09:45.471488 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:45.471449 6735 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 04:09:45.471782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:45.471753 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 04:09:46.003314 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:46.003263 6735 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c10c88c_389b_41b7_be99_a1dd5655fd4e.slice/crio-8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c10c88c_389b_41b7_be99_a1dd5655fd4e.slice/crio-conmon-8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f.scope\": RecentStats: unable to find data in memory cache]" Nov 01 04:09:46.277299 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.277267 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:09:46.326485 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326453 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-data\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326671 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326501 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istiod-ca-cert\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326671 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326536 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-certs\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326671 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326598 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-envoy\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326671 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326642 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-socket\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326686 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-kube-api-access-pbwfp\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-kube-api-access-pbwfp\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326727 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-credential-socket\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326765 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-podinfo\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326805 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-data" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:09:46.326939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326909 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-token\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-token\") pod \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\" (UID: \"1c10c88c-389b-41b7-be99-a1dd5655fd4e\") " Nov 01 04:09:46.326939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.326916 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-certs" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:09:46.327265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327011 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-socket" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:09:46.327265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327065 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istiod-ca-cert" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:09:46.327265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327240 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-credential-socket" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:09:46.327936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327830 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.327936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327866 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.327936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327883 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.327936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327899 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.327936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.327913 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.330066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.329688 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-podinfo" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:09:46.330066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.329809 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-envoy" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:09:46.330066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.329832 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-token" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:09:46.330066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.329903 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-kube-api-access-pbwfp" pod "1c10c88c-389b-41b7-be99-a1dd5655fd4e" (UID: "1c10c88c-389b-41b7-be99-a1dd5655fd4e"). InnerVolumeSpecName "kube-api-access-pbwfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:09:46.429353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.429308 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.429353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.429346 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbwfp\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-kube-api-access-pbwfp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.429353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.429361 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.429595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.429375 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c10c88c-389b-41b7-be99-a1dd5655fd4e-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:46.475580 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.475532 6735 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 04:09:46.475783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:46.475599 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 04:09:47.008624 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.008587 6735 generic.go:356] "Generic (PLEG): container finished" podID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerID="8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f" exitCode=0 Nov 01 04:09:47.009140 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.008632 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" event={"ID":"1c10c88c-389b-41b7-be99-a1dd5655fd4e","Type":"ContainerDied","Data":"8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f"} Nov 01 04:09:47.009140 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.008686 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" Nov 01 04:09:47.009140 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.008725 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr" event={"ID":"1c10c88c-389b-41b7-be99-a1dd5655fd4e","Type":"ContainerDied","Data":"cfafbdd2a2d81f77641911cbc5578ad4ccb95a9a058865b1f7faf433923ed18c"} Nov 01 04:09:47.009140 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.008755 6735 scope.go:122] "RemoveContainer" containerID="8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f" Nov 01 04:09:47.019622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.019607 6735 scope.go:122] "RemoveContainer" containerID="91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda" Nov 01 04:09:47.030505 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.030480 6735 scope.go:122] "RemoveContainer" containerID="8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f" Nov 01 04:09:47.030799 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:47.030776 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f\": container with ID starting with 8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f not found: ID does not exist" containerID="8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f" Nov 01 04:09:47.030890 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.030814 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f"} err="failed to get container status \"8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f\": rpc error: code = NotFound desc = could not find container \"8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f\": container with ID starting with 8ea64a863fff38d0a0cbc11f0d03e83d06fe5a371c73515bb627e424fd7bbf4f not found: ID does not exist" Nov 01 04:09:47.030890 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.030841 6735 scope.go:122] "RemoveContainer" containerID="91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda" Nov 01 04:09:47.031151 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:47.031121 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda\": container with ID starting with 91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda not found: ID does not exist" containerID="91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda" Nov 01 04:09:47.031258 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.031159 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda"} err="failed to get container status \"91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda\": rpc error: code = NotFound desc = could not find container \"91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda\": container with ID starting with 91adb6e1a88643897ad3f28f2a6baf5f0d382494da3fb6846d859b09d3a8fbda not found: ID does not exist" Nov 01 04:09:47.173222 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.173183 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr"] Nov 01 04:09:47.177927 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.177895 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-aab51f-openshift-default-8589c8cbbf-l4hvr"] Nov 01 04:09:47.475592 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.475563 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:09:47.476592 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:47.476575 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:09:48.526528 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:48.526488 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" path="/var/lib/kubelet/pods/1c10c88c-389b-41b7-be99-a1dd5655fd4e/volumes" Nov 01 04:09:53.583463 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:53.583425 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl"] Nov 01 04:09:58.584385 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:58.584342 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" podUID="c3ca7817-a2c1-49b2-84b7-3777f0206fae" containerName="payload-processing" containerID="cri-o://aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b" gracePeriod=25 Nov 01 04:09:59.028515 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.028486 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:09:59.054708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.054677 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:09:59.104772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.104732 6735 generic.go:356] "Generic (PLEG): container finished" podID="c3ca7817-a2c1-49b2-84b7-3777f0206fae" containerID="aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b" exitCode=0 Nov 01 04:09:59.104919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.104781 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" Nov 01 04:09:59.104919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.104808 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" event={"ID":"c3ca7817-a2c1-49b2-84b7-3777f0206fae","Type":"ContainerDied","Data":"aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b"} Nov 01 04:09:59.104919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.104858 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl" event={"ID":"c3ca7817-a2c1-49b2-84b7-3777f0206fae","Type":"ContainerDied","Data":"c6990deba7f1ec1254d25edba0254e51c81787fb154c1ea4da9826743fc50986"} Nov 01 04:09:59.104919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.104880 6735 scope.go:122] "RemoveContainer" containerID="aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b" Nov 01 04:09:59.114613 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.114598 6735 scope.go:122] "RemoveContainer" containerID="aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b" Nov 01 04:09:59.114880 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:09:59.114860 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b\": container with ID starting with aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b not found: ID does not exist" containerID="aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b" Nov 01 04:09:59.114919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.114889 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b"} err="failed to get container status \"aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b\": rpc error: code = NotFound desc = could not find container \"aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b\": container with ID starting with aaf58cee82bbb42229c68e2743d25f3d79002e4a39092f32e94ca878997de45b not found: ID does not exist" Nov 01 04:09:59.133445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.133414 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ca7817-a2c1-49b2-84b7-3777f0206fae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ca7817-a2c1-49b2-84b7-3777f0206fae-plugins-config-volume\") pod \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " Nov 01 04:09:59.133773 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.133753 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ca7817-a2c1-49b2-84b7-3777f0206fae-kube-api-access-xwz4r\" (UniqueName: \"kubernetes.io/projected/c3ca7817-a2c1-49b2-84b7-3777f0206fae-kube-api-access-xwz4r\") pod \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\" (UID: \"c3ca7817-a2c1-49b2-84b7-3777f0206fae\") " Nov 01 04:09:59.133867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.133847 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ca7817-a2c1-49b2-84b7-3777f0206fae-plugins-config-volume" pod "c3ca7817-a2c1-49b2-84b7-3777f0206fae" (UID: "c3ca7817-a2c1-49b2-84b7-3777f0206fae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:09:59.134254 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.134233 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3ca7817-a2c1-49b2-84b7-3777f0206fae-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:59.136187 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.136162 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ca7817-a2c1-49b2-84b7-3777f0206fae-kube-api-access-xwz4r" pod "c3ca7817-a2c1-49b2-84b7-3777f0206fae" (UID: "c3ca7817-a2c1-49b2-84b7-3777f0206fae"). InnerVolumeSpecName "kube-api-access-xwz4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:09:59.235400 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.235354 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwz4r\" (UniqueName: \"kubernetes.io/projected/c3ca7817-a2c1-49b2-84b7-3777f0206fae-kube-api-access-xwz4r\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:09:59.522850 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.522764 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl"] Nov 01 04:09:59.568811 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:09:59.568782 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6dd36b-5b4bb5b5dc-hklbl"] Nov 01 04:10:00.528837 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:00.528799 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ca7817-a2c1-49b2-84b7-3777f0206fae" path="/var/lib/kubelet/pods/c3ca7817-a2c1-49b2-84b7-3777f0206fae/volumes" Nov 01 04:10:03.001981 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:10:03.001940 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:10:07.584425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:07.584381 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7"] Nov 01 04:10:12.585018 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:12.584975 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" podUID="4ddb652f-13af-4a31-86d0-331f531e8ccc" containerName="payload-pre-processing" containerID="cri-o://4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8" gracePeriod=25 Nov 01 04:10:12.853445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:12.853376 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:10:12.947381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:12.947347 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddb652f-13af-4a31-86d0-331f531e8ccc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddb652f-13af-4a31-86d0-331f531e8ccc-plugins-config-volume\") pod \"4ddb652f-13af-4a31-86d0-331f531e8ccc\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " Nov 01 04:10:12.947381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:12.947386 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddb652f-13af-4a31-86d0-331f531e8ccc-kube-api-access-644cq\" (UniqueName: \"kubernetes.io/projected/4ddb652f-13af-4a31-86d0-331f531e8ccc-kube-api-access-644cq\") pod \"4ddb652f-13af-4a31-86d0-331f531e8ccc\" (UID: \"4ddb652f-13af-4a31-86d0-331f531e8ccc\") " Nov 01 04:10:12.947710 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:12.947687 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddb652f-13af-4a31-86d0-331f531e8ccc-plugins-config-volume" pod "4ddb652f-13af-4a31-86d0-331f531e8ccc" (UID: "4ddb652f-13af-4a31-86d0-331f531e8ccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:10:12.947879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:12.947854 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddb652f-13af-4a31-86d0-331f531e8ccc-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:12.949602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:12.949576 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddb652f-13af-4a31-86d0-331f531e8ccc-kube-api-access-644cq" pod "4ddb652f-13af-4a31-86d0-331f531e8ccc" (UID: "4ddb652f-13af-4a31-86d0-331f531e8ccc"). InnerVolumeSpecName "kube-api-access-644cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:10:13.049086 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.049020 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-644cq\" (UniqueName: \"kubernetes.io/projected/4ddb652f-13af-4a31-86d0-331f531e8ccc-kube-api-access-644cq\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:13.222580 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.222549 6735 generic.go:356] "Generic (PLEG): container finished" podID="4ddb652f-13af-4a31-86d0-331f531e8ccc" containerID="4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8" exitCode=0 Nov 01 04:10:13.222746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.222596 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" Nov 01 04:10:13.222746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.222599 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" event={"ID":"4ddb652f-13af-4a31-86d0-331f531e8ccc","Type":"ContainerDied","Data":"4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8"} Nov 01 04:10:13.222746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.222721 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7" event={"ID":"4ddb652f-13af-4a31-86d0-331f531e8ccc","Type":"ContainerDied","Data":"3e6f48a6effbb14dfb0893a331c7468ef677203b976a3cb44efde8297d238e21"} Nov 01 04:10:13.222865 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.222754 6735 scope.go:122] "RemoveContainer" containerID="4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8" Nov 01 04:10:13.233271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.233251 6735 scope.go:122] "RemoveContainer" containerID="4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8" Nov 01 04:10:13.233519 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:10:13.233496 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8\": container with ID starting with 4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8 not found: ID does not exist" containerID="4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8" Nov 01 04:10:13.233566 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.233528 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8"} err="failed to get container status \"4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8\": rpc error: code = NotFound desc = could not find container \"4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8\": container with ID starting with 4f9232fc3a0ad67341fffa52cd106497296e040ec44548b2d22d010911071cf8 not found: ID does not exist" Nov 01 04:10:13.377432 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.377398 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7"] Nov 01 04:10:13.423652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:13.423624 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78dad9-577bbf676c-4m7w7"] Nov 01 04:10:14.526754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:14.526724 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddb652f-13af-4a31-86d0-331f531e8ccc" path="/var/lib/kubelet/pods/4ddb652f-13af-4a31-86d0-331f531e8ccc/volumes" Nov 01 04:10:26.500497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.500466 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj"] Nov 01 04:10:26.501160 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501143 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" Nov 01 04:10:26.501217 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501162 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" Nov 01 04:10:26.501217 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501183 6735 cpu_manager.go:414] "containerMap: removing container" podUID="10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" containerName="maas-api" Nov 01 04:10:26.501217 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501187 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" containerName="maas-api" Nov 01 04:10:26.501313 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501218 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4ddb652f-13af-4a31-86d0-331f531e8ccc" containerName="payload-pre-processing" Nov 01 04:10:26.501313 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501228 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddb652f-13af-4a31-86d0-331f531e8ccc" containerName="payload-pre-processing" Nov 01 04:10:26.501313 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501239 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c3ca7817-a2c1-49b2-84b7-3777f0206fae" containerName="payload-processing" Nov 01 04:10:26.501313 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501245 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ca7817-a2c1-49b2-84b7-3777f0206fae" containerName="payload-processing" Nov 01 04:10:26.501429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501343 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" Nov 01 04:10:26.501429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501353 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e5bcb3-80b0-4aa0-a000-0ab53b6cc276" containerName="maas-api" Nov 01 04:10:26.501429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501366 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddb652f-13af-4a31-86d0-331f531e8ccc" containerName="payload-pre-processing" Nov 01 04:10:26.501429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.501373 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ca7817-a2c1-49b2-84b7-3777f0206fae" containerName="payload-processing" Nov 01 04:10:26.505028 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.505006 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.554232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.554200 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-078e0d65-openshift-default-dockercfg-dr25l\"" Nov 01 04:10:26.604237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.604201 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj"] Nov 01 04:10:26.628769 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.628731 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.628959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.628807 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c210ff5-234b-495f-835e-768b3f5a5b88-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.628959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.628938 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq9ps\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-kube-api-access-pq9ps\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.629118 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.628996 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.629118 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.629028 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.629232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.629122 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.629232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.629168 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.629335 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.629233 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.629335 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.629329 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.731121 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.731090 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq9ps\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-kube-api-access-pq9ps\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.731301 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.731138 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.731368 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.731319 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.731445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.731423 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.731687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.731661 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.731825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.731741 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.731825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.731766 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.732469 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.732111 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.732469 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.732121 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.732469 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.732231 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.732469 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.732302 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c210ff5-234b-495f-835e-768b3f5a5b88-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.732469 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.732430 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.732791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.732738 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.733094 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.733029 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c210ff5-234b-495f-835e-768b3f5a5b88-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.733881 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.733851 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.734340 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.734320 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.789365 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.789293 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.789495 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.789432 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq9ps\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-kube-api-access-pq9ps\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:26.819461 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:26.819430 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:27.239467 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.239435 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj"] Nov 01 04:10:27.240361 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:10:27.240332 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c210ff5_234b_495f_835e_768b3f5a5b88.slice/crio-373548ce25fb204eea6643dd0489a3e2b66c238e336d957f17af88d4f06bad9c WatchSource:0}: Error finding container 373548ce25fb204eea6643dd0489a3e2b66c238e336d957f17af88d4f06bad9c: Status 404 returned error can't find the container with id 373548ce25fb204eea6643dd0489a3e2b66c238e336d957f17af88d4f06bad9c Nov 01 04:10:27.242989 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.242955 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:27.243068 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.243020 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:27.243114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.243069 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:27.323622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.323576 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" event={"ID":"3c210ff5-234b-495f-835e-768b3f5a5b88","Type":"ContainerStarted","Data":"5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc"} Nov 01 04:10:27.323995 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.323744 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" event={"ID":"3c210ff5-234b-495f-835e-768b3f5a5b88","Type":"ContainerStarted","Data":"373548ce25fb204eea6643dd0489a3e2b66c238e336d957f17af88d4f06bad9c"} Nov 01 04:10:27.440006 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.439939 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podStartSLOduration=1.439917819 podStartE2EDuration="1.439917819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:27.437405646 +0000 UTC m=+1639.026500276" watchObservedRunningTime="2026-11-01 04:10:27.439917819 +0000 UTC m=+1639.029012449" Nov 01 04:10:27.820202 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:27.820171 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:28.821784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:28.821749 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 04:10:28.822216 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:28.821804 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 04:10:29.819765 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:29.819731 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 04:10:29.819965 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:29.819786 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 04:10:30.820162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:30.820129 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 04:10:30.820590 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:30.820190 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 04:10:31.820490 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:31.820457 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 04:10:31.820881 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:31.820510 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 04:10:32.819932 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:32.819883 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 04:10:32.820121 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:32.819945 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 04:10:33.082350 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:10:33.082252 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:10:33.820850 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:33.820805 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 04:10:33.821075 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:33.820875 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 04:10:34.823936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:34.823907 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:34.824433 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:34.824323 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:34.825019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:34.825004 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:10:37.087296 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.087245 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-e1bc22"] Nov 01 04:10:37.087969 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.087952 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" Nov 01 04:10:37.088016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.087971 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" Nov 01 04:10:37.088124 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.088115 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c10c88c-389b-41b7-be99-a1dd5655fd4e" containerName="istio-proxy" Nov 01 04:10:37.092855 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.092821 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:37.122607 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.122568 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvddz\" (UniqueName: \"kubernetes.io/projected/0ce20265-1756-4c3d-a0dc-97e139be5793-kube-api-access-bvddz\") pod \"test-curl-7422-e1bc22\" (UID: \"0ce20265-1756-4c3d-a0dc-97e139be5793\") " pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:37.144714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.144684 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-e1bc22"] Nov 01 04:10:37.187524 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.187496 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zjjms\"" Nov 01 04:10:37.223747 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.223706 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvddz\" (UniqueName: \"kubernetes.io/projected/0ce20265-1756-4c3d-a0dc-97e139be5793-kube-api-access-bvddz\") pod \"test-curl-7422-e1bc22\" (UID: \"0ce20265-1756-4c3d-a0dc-97e139be5793\") " pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:37.282151 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.282123 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvddz\" (UniqueName: \"kubernetes.io/projected/0ce20265-1756-4c3d-a0dc-97e139be5793-kube-api-access-bvddz\") pod \"test-curl-7422-e1bc22\" (UID: \"0ce20265-1756-4c3d-a0dc-97e139be5793\") " pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:37.408473 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.408394 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:37.776516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:37.776486 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-e1bc22"] Nov 01 04:10:37.777625 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:10:37.777596 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ce20265_1756_4c3d_a0dc_97e139be5793.slice/crio-3b71ea707594e075d5898d9e96c7372c04c038c4cb9df9707411ca32d856803e WatchSource:0}: Error finding container 3b71ea707594e075d5898d9e96c7372c04c038c4cb9df9707411ca32d856803e: Status 404 returned error can't find the container with id 3b71ea707594e075d5898d9e96c7372c04c038c4cb9df9707411ca32d856803e Nov 01 04:10:38.428746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:38.428714 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-e1bc22" event={"ID":"0ce20265-1756-4c3d-a0dc-97e139be5793","Type":"ContainerStarted","Data":"3b71ea707594e075d5898d9e96c7372c04c038c4cb9df9707411ca32d856803e"} Nov 01 04:10:41.093594 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.093561 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8"] Nov 01 04:10:41.094006 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.093769 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" podUID="8562485e-8985-4042-a0c2-1b0c1f0193d9" containerName="maas-api" containerID="cri-o://c244467a0f52da67d45a2b96074c0c96bcd901f98d4d5d5764c36bc12f092bd3" gracePeriod=30 Nov 01 04:10:41.453887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.453853 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-e1bc22" event={"ID":"0ce20265-1756-4c3d-a0dc-97e139be5793","Type":"ContainerStarted","Data":"663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b"} Nov 01 04:10:41.456097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.456053 6735 generic.go:356] "Generic (PLEG): container finished" podID="8562485e-8985-4042-a0c2-1b0c1f0193d9" containerID="c244467a0f52da67d45a2b96074c0c96bcd901f98d4d5d5764c36bc12f092bd3" exitCode=0 Nov 01 04:10:41.456097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.456068 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" event={"ID":"8562485e-8985-4042-a0c2-1b0c1f0193d9","Type":"ContainerDied","Data":"c244467a0f52da67d45a2b96074c0c96bcd901f98d4d5d5764c36bc12f092bd3"} Nov 01 04:10:41.456305 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.456143 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" event={"ID":"8562485e-8985-4042-a0c2-1b0c1f0193d9","Type":"ContainerDied","Data":"e77e41e94b866e3a885c97265603e693476bedfcc94f1dfb8dfd05fbf2d91ff5"} Nov 01 04:10:41.456305 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.456168 6735 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77e41e94b866e3a885c97265603e693476bedfcc94f1dfb8dfd05fbf2d91ff5" Nov 01 04:10:41.457579 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.457558 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:10:41.559588 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.559536 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8562485e-8985-4042-a0c2-1b0c1f0193d9-openshift-service-ca" pod "8562485e-8985-4042-a0c2-1b0c1f0193d9" (UID: "8562485e-8985-4042-a0c2-1b0c1f0193d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:10:41.559845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.559822 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8562485e-8985-4042-a0c2-1b0c1f0193d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8562485e-8985-4042-a0c2-1b0c1f0193d9-openshift-service-ca\") pod \"8562485e-8985-4042-a0c2-1b0c1f0193d9\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " Nov 01 04:10:41.560136 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.560111 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8562485e-8985-4042-a0c2-1b0c1f0193d9-kube-api-access-w9sc5\" (UniqueName: \"kubernetes.io/projected/8562485e-8985-4042-a0c2-1b0c1f0193d9-kube-api-access-w9sc5\") pod \"8562485e-8985-4042-a0c2-1b0c1f0193d9\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " Nov 01 04:10:41.568589 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.560297 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-metrics-certs\") pod \"8562485e-8985-4042-a0c2-1b0c1f0193d9\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " Nov 01 04:10:41.568941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.568917 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls\") pod \"8562485e-8985-4042-a0c2-1b0c1f0193d9\" (UID: \"8562485e-8985-4042-a0c2-1b0c1f0193d9\") " Nov 01 04:10:41.606499 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.604722 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8562485e-8985-4042-a0c2-1b0c1f0193d9-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:41.614106 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.607646 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk"] Nov 01 04:10:41.614106 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.608654 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls" pod "8562485e-8985-4042-a0c2-1b0c1f0193d9" (UID: "8562485e-8985-4042-a0c2-1b0c1f0193d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:10:41.614106 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.610327 6735 cpu_manager.go:414] "containerMap: removing container" podUID="8562485e-8985-4042-a0c2-1b0c1f0193d9" containerName="maas-api" Nov 01 04:10:41.614106 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.610348 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8562485e-8985-4042-a0c2-1b0c1f0193d9" containerName="maas-api" Nov 01 04:10:41.614106 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.610733 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="8562485e-8985-4042-a0c2-1b0c1f0193d9" containerName="maas-api" Nov 01 04:10:41.642427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.624667 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.682742 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.662941 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8562485e-8985-4042-a0c2-1b0c1f0193d9-kube-api-access-w9sc5" pod "8562485e-8985-4042-a0c2-1b0c1f0193d9" (UID: "8562485e-8985-4042-a0c2-1b0c1f0193d9"). InnerVolumeSpecName "kube-api-access-w9sc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:10:41.682742 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.665854 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-metrics-certs" pod "8562485e-8985-4042-a0c2-1b0c1f0193d9" (UID: "8562485e-8985-4042-a0c2-1b0c1f0193d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:10:41.682742 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.681547 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg"] Nov 01 04:10:41.697098 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.695542 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk"] Nov 01 04:10:41.697098 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.695588 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg"] Nov 01 04:10:41.697098 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.695682 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.704974 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.704920 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7422-e1bc22" podStartSLOduration=1.977990784 podStartE2EDuration="4.704903685s" totalImagesPullingTime="2.726912901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:41.704688047 +0000 UTC m=+1653.293782677" watchObservedRunningTime="2026-11-01 04:10:41.704903685 +0000 UTC m=+1653.293998315" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.707951 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.708018 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.708080 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgs44\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-kube-api-access-lgs44\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.708146 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.708214 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.708245 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.708335 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.708548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.708385 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.709868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.709829 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a37e982-6885-4ad9-a0f9-7604756f6612-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.710103 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.710080 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9sc5\" (UniqueName: \"kubernetes.io/projected/8562485e-8985-4042-a0c2-1b0c1f0193d9-kube-api-access-w9sc5\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:41.710182 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.710113 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:41.710182 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.710129 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8562485e-8985-4042-a0c2-1b0c1f0193d9-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:41.811223 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811190 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a37e982-6885-4ad9-a0f9-7604756f6612-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811244 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.811406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811320 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.811406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811383 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.811577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811407 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4zf\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-kube-api-access-sz4zf\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.811577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811438 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811554 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811739 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811602 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs44\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-kube-api-access-lgs44\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811739 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811665 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811739 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811701 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902b46b1-e745-435e-bbfa-94735bee93bf-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.811739 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811705 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811807 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811843 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811860 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.811941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811918 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902b46b1-e745-435e-bbfa-94735bee93bf-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.812171 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.811961 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a37e982-6885-4ad9-a0f9-7604756f6612-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.812171 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.812017 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.812171 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.812099 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.812171 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.812131 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.812381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.812208 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.812381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.812247 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.812648 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.812382 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.812648 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.812500 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.813940 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.813917 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.814333 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.814318 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.869476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.869444 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.869665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.869647 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgs44\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-kube-api-access-lgs44\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:41.913137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913102 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902b46b1-e745-435e-bbfa-94735bee93bf-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913161 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913203 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913262 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913414 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913452 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913606 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913565 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-certs\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913674 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913610 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913674 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913587 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913674 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913663 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4zf\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-kube-api-access-sz4zf\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913675 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-credential-socket\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913768 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902b46b1-e745-435e-bbfa-94735bee93bf-istiod-ca-cert\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913772 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902b46b1-e745-435e-bbfa-94735bee93bf-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.913808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.913797 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-data\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.915689 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.915664 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-envoy\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.915942 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.915924 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902b46b1-e745-435e-bbfa-94735bee93bf-istio-podinfo\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.971502 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.971468 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-istio-token\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:41.971711 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:41.971686 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4zf\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-kube-api-access-sz4zf\") pod \"e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:42.001352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.001322 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:42.017381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.017361 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:42.461988 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.461960 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8" Nov 01 04:10:42.591712 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.591682 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk"] Nov 01 04:10:42.592413 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:10:42.592387 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a37e982_6885_4ad9_a0f9_7604756f6612.slice/crio-d2119309245bbeb546e54b6ffcd2cabe5c426ea6a1f5d2fadc507a37d637e5c2 WatchSource:0}: Error finding container d2119309245bbeb546e54b6ffcd2cabe5c426ea6a1f5d2fadc507a37d637e5c2: Status 404 returned error can't find the container with id d2119309245bbeb546e54b6ffcd2cabe5c426ea6a1f5d2fadc507a37d637e5c2 Nov 01 04:10:42.595194 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.595159 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:42.595285 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.595232 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:42.595285 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.595264 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:42.610368 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.610334 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg"] Nov 01 04:10:42.617542 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.617518 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8"] Nov 01 04:10:42.624620 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:10:42.624583 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902b46b1_e745_435e_bbfa_94735bee93bf.slice/crio-0831679dbfdc194cedfc397459efa6178c28cd7d715992ea9ad715c210d7293e WatchSource:0}: Error finding container 0831679dbfdc194cedfc397459efa6178c28cd7d715992ea9ad715c210d7293e: Status 404 returned error can't find the container with id 0831679dbfdc194cedfc397459efa6178c28cd7d715992ea9ad715c210d7293e Nov 01 04:10:42.627697 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.627663 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:42.627779 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.627742 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:42.627822 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.627782 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:10:42.638627 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:42.638599 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3344e8-54886b9d6f-lh4t8"] Nov 01 04:10:43.279169 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.279134 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-e1bc22"] Nov 01 04:10:43.294060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.290098 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-e1bc22"] Nov 01 04:10:43.504923 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.504882 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" event={"ID":"1a37e982-6885-4ad9-a0f9-7604756f6612","Type":"ContainerStarted","Data":"38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249"} Nov 01 04:10:43.505398 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.504946 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" event={"ID":"1a37e982-6885-4ad9-a0f9-7604756f6612","Type":"ContainerStarted","Data":"d2119309245bbeb546e54b6ffcd2cabe5c426ea6a1f5d2fadc507a37d637e5c2"} Nov 01 04:10:43.506731 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.506704 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" event={"ID":"902b46b1-e745-435e-bbfa-94735bee93bf","Type":"ContainerStarted","Data":"e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1"} Nov 01 04:10:43.506792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.506758 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" event={"ID":"902b46b1-e745-435e-bbfa-94735bee93bf","Type":"ContainerStarted","Data":"0831679dbfdc194cedfc397459efa6178c28cd7d715992ea9ad715c210d7293e"} Nov 01 04:10:43.506843 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.506783 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7422-e1bc22" podUID="0ce20265-1756-4c3d-a0dc-97e139be5793" containerName="test-curl-7422-e1bc22" containerID="cri-o://663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b" gracePeriod=2 Nov 01 04:10:43.634418 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.634224 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podStartSLOduration=2.63421227 podStartE2EDuration="2.63421227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:43.633139774 +0000 UTC m=+1655.222234403" watchObservedRunningTime="2026-11-01 04:10:43.63421227 +0000 UTC m=+1655.223306900" Nov 01 04:10:43.687058 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:10:43.686361 6735 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7422-e1bc22\" is forbidden: User \"system:node:ip-10-0-128-69.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-69.us-west-2.compute.internal' and this object" podUID="0ce20265-1756-4c3d-a0dc-97e139be5793" pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:43.802254 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:43.802195 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podStartSLOduration=2.802177601 podStartE2EDuration="2.802177601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:43.800638971 +0000 UTC m=+1655.389733600" watchObservedRunningTime="2026-11-01 04:10:43.802177601 +0000 UTC m=+1655.391272230" Nov 01 04:10:44.002712 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:44.002662 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:44.003760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:44.003726 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 04:10:44.003887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:44.003787 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 04:10:44.018287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:44.018261 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:44.019601 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:44.019577 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 04:10:44.019710 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:44.019626 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 04:10:44.527676 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:44.527641 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8562485e-8985-4042-a0c2-1b0c1f0193d9" path="/var/lib/kubelet/pods/8562485e-8985-4042-a0c2-1b0c1f0193d9/volumes" Nov 01 04:10:45.002193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:45.002148 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 04:10:45.002390 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:45.002219 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 04:10:45.018156 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:45.018126 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 04:10:45.018365 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:45.018339 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 04:10:45.842141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:45.842116 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:45.935688 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:45.935657 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce20265-1756-4c3d-a0dc-97e139be5793-kube-api-access-bvddz\" (UniqueName: \"kubernetes.io/projected/0ce20265-1756-4c3d-a0dc-97e139be5793-kube-api-access-bvddz\") pod \"0ce20265-1756-4c3d-a0dc-97e139be5793\" (UID: \"0ce20265-1756-4c3d-a0dc-97e139be5793\") " Nov 01 04:10:45.939734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:45.939709 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce20265-1756-4c3d-a0dc-97e139be5793-kube-api-access-bvddz" pod "0ce20265-1756-4c3d-a0dc-97e139be5793" (UID: "0ce20265-1756-4c3d-a0dc-97e139be5793"). InnerVolumeSpecName "kube-api-access-bvddz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:10:46.002624 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.002586 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 04:10:46.002802 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.002676 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 04:10:46.018180 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.018149 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 04:10:46.018300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.018191 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 04:10:46.037590 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.037566 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvddz\" (UniqueName: \"kubernetes.io/projected/0ce20265-1756-4c3d-a0dc-97e139be5793-kube-api-access-bvddz\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:46.528127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.528090 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce20265-1756-4c3d-a0dc-97e139be5793" path="/var/lib/kubelet/pods/0ce20265-1756-4c3d-a0dc-97e139be5793/volumes" Nov 01 04:10:46.530118 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.530094 6735 generic.go:356] "Generic (PLEG): container finished" podID="0ce20265-1756-4c3d-a0dc-97e139be5793" containerID="663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b" exitCode=137 Nov 01 04:10:46.530259 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.530134 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-e1bc22" Nov 01 04:10:46.530259 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.530195 6735 scope.go:122] "RemoveContainer" containerID="663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b" Nov 01 04:10:46.541143 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.541125 6735 scope.go:122] "RemoveContainer" containerID="663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b" Nov 01 04:10:46.541528 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:10:46.541454 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b\": container with ID starting with 663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b not found: ID does not exist" containerID="663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b" Nov 01 04:10:46.541528 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:46.541487 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b"} err="failed to get container status \"663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b\": rpc error: code = NotFound desc = could not find container \"663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b\": container with ID starting with 663a105cef037c2fa384fc19b03b092840fb0874f68ca41f63f31ccce15fd29b not found: ID does not exist" Nov 01 04:10:47.001642 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:47.001605 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 04:10:47.002170 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:47.001665 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 04:10:47.017749 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:47.017715 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 04:10:47.017901 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:47.017758 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 04:10:48.002074 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:48.002018 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 04:10:48.002481 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:48.002122 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 04:10:48.017876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:48.017840 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 04:10:48.017976 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:48.017935 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 04:10:49.001951 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:49.001907 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 01 04:10:49.002149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:49.001982 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 01 04:10:49.018437 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:49.018399 6735 patch_prober.go:29] interesting pod/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 01 04:10:49.018599 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:49.018468 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 01 04:10:50.037114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.037082 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:50.037601 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.037180 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:50.038914 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.038224 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:50.038914 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.038511 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:50.039488 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.039467 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:10:50.039570 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.039518 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:10:50.245525 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.245483 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr"] Nov 01 04:10:50.246079 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.246019 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" containerID="cri-o://37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574" gracePeriod=30 Nov 01 04:10:50.390109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.389285 6735 patch_prober.go:29] interesting pod/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:10:50.390109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:50.389346 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:10:54.059376 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:54.059343 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:10:55.537928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.537854 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:10:55.602349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602313 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-socket\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602354 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-data\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602388 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-certs\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602409 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-credential-socket\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602462 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-kube-api-access-9tnkh\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-kube-api-access-9tnkh\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602495 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istiod-ca-cert\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602536 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-podinfo\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602595 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-token\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-token\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602607 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-socket" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:10:55.602722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602675 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-envoy\") pod \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\" (UID: \"47420ed3-26f3-4302-ad0d-2e41e9a94a7c\") " Nov 01 04:10:55.602860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602703 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-data" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:10:55.602860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602790 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-credential-socket" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:10:55.602997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.602974 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istiod-ca-cert" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:10:55.603439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.603332 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-certs" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:10:55.603439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.603333 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.603439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.603375 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.603439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.603393 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.603439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.603409 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.604987 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.604964 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-envoy" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:10:55.605391 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.605374 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-podinfo" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:10:55.605542 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.605526 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-token" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:10:55.605635 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.605616 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-kube-api-access-9tnkh" pod "47420ed3-26f3-4302-ad0d-2e41e9a94a7c" (UID: "47420ed3-26f3-4302-ad0d-2e41e9a94a7c"). InnerVolumeSpecName "kube-api-access-9tnkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:10:55.649551 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.649520 6735 generic.go:356] "Generic (PLEG): container finished" podID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerID="37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574" exitCode=0 Nov 01 04:10:55.649683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.649588 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" event={"ID":"47420ed3-26f3-4302-ad0d-2e41e9a94a7c","Type":"ContainerDied","Data":"37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574"} Nov 01 04:10:55.649683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.649649 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" event={"ID":"47420ed3-26f3-4302-ad0d-2e41e9a94a7c","Type":"ContainerDied","Data":"aeb3400e270a754fc669f992969ccae3ed3363a8666033194531cc9f45ca971f"} Nov 01 04:10:55.649683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.649585 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr" Nov 01 04:10:55.649796 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.649671 6735 scope.go:122] "RemoveContainer" containerID="37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574" Nov 01 04:10:55.660604 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.660588 6735 scope.go:122] "RemoveContainer" containerID="ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e" Nov 01 04:10:55.669524 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.669505 6735 scope.go:122] "RemoveContainer" containerID="37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574" Nov 01 04:10:55.669777 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:10:55.669746 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574\": container with ID starting with 37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574 not found: ID does not exist" containerID="37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574" Nov 01 04:10:55.669839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.669775 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574"} err="failed to get container status \"37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574\": rpc error: code = NotFound desc = could not find container \"37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574\": container with ID starting with 37262788086f4b4ac35700cca5ccdbfc1a0bd3d03643dd12c47436fac84c5574 not found: ID does not exist" Nov 01 04:10:55.669839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.669796 6735 scope.go:122] "RemoveContainer" containerID="ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e" Nov 01 04:10:55.670133 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:10:55.670106 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e\": container with ID starting with ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e not found: ID does not exist" containerID="ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e" Nov 01 04:10:55.670205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.670143 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e"} err="failed to get container status \"ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e\": rpc error: code = NotFound desc = could not find container \"ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e\": container with ID starting with ebe6df93608259ef0f7ed088cfc9fa6c004e0826cc5269ecfa93759ca2dc304e not found: ID does not exist" Nov 01 04:10:55.704614 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.704576 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.704614 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.704614 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.704791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.704631 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnkh\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-kube-api-access-9tnkh\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.704791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.704645 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.704791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.704657 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47420ed3-26f3-4302-ad0d-2e41e9a94a7c-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:10:55.804897 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.804811 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr"] Nov 01 04:10:55.811188 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:55.811165 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3344e8-openshift-default-7cd764b965-c7tcr"] Nov 01 04:10:56.373239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:56.373203 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz"] Nov 01 04:10:56.468277 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:56.468245 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf"] Nov 01 04:10:56.527984 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:10:56.527947 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" path="/var/lib/kubelet/pods/47420ed3-26f3-4302-ad0d-2e41e9a94a7c/volumes" Nov 01 04:11:00.715188 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.715145 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6"] Nov 01 04:11:00.722061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.716123 6735 cpu_manager.go:414] "containerMap: removing container" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" Nov 01 04:11:00.747064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.738564 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" Nov 01 04:11:00.747064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.738689 6735 cpu_manager.go:414] "containerMap: removing container" podUID="0ce20265-1756-4c3d-a0dc-97e139be5793" containerName="test-curl-7422-e1bc22" Nov 01 04:11:00.747064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.738702 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce20265-1756-4c3d-a0dc-97e139be5793" containerName="test-curl-7422-e1bc22" Nov 01 04:11:00.747064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.738964 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" Nov 01 04:11:00.747064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.738992 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce20265-1756-4c3d-a0dc-97e139be5793" containerName="test-curl-7422-e1bc22" Nov 01 04:11:00.760186 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.757151 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:00.804889 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.771926 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6"] Nov 01 04:11:00.888177 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.831117 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-157fed\"" Nov 01 04:11:00.888177 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.837181 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9gn\" (UniqueName: \"kubernetes.io/projected/aeccb9e1-5fd7-457b-a11f-84f8baa26120-kube-api-access-sx9gn\") pod \"payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:00.888460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.888412 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeccb9e1-5fd7-457b-a11f-84f8baa26120-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:00.888708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.886873 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-157fed-dockercfg-hxfgw\"" Nov 01 04:11:00.889926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.889876 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474"] Nov 01 04:11:00.891136 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.891105 6735 cpu_manager.go:414] "containerMap: removing container" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" Nov 01 04:11:00.891227 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.891145 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" Nov 01 04:11:00.891396 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.891382 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="47420ed3-26f3-4302-ad0d-2e41e9a94a7c" containerName="istio-proxy" Nov 01 04:11:00.895759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.895741 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:00.935403 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.935375 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474"] Nov 01 04:11:00.989846 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.989770 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9gn\" (UniqueName: \"kubernetes.io/projected/aeccb9e1-5fd7-457b-a11f-84f8baa26120-kube-api-access-sx9gn\") pod \"payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:00.989846 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.989817 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ndd\" (UniqueName: \"kubernetes.io/projected/643eea33-f1c4-438b-a453-5ed690d5ca96-kube-api-access-b6ndd\") pod \"payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:00.990021 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.989859 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643eea33-f1c4-438b-a453-5ed690d5ca96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:00.990021 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.989959 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeccb9e1-5fd7-457b-a11f-84f8baa26120-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:00.990688 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:00.990662 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeccb9e1-5fd7-457b-a11f-84f8baa26120-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:01.051745 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.051711 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9gn\" (UniqueName: \"kubernetes.io/projected/aeccb9e1-5fd7-457b-a11f-84f8baa26120-kube-api-access-sx9gn\") pod \"payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:01.091593 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.091560 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ndd\" (UniqueName: \"kubernetes.io/projected/643eea33-f1c4-438b-a453-5ed690d5ca96-kube-api-access-b6ndd\") pod \"payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:01.091750 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.091630 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643eea33-f1c4-438b-a453-5ed690d5ca96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:01.092359 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.092339 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643eea33-f1c4-438b-a453-5ed690d5ca96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:01.149434 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.149398 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ndd\" (UniqueName: \"kubernetes.io/projected/643eea33-f1c4-438b-a453-5ed690d5ca96-kube-api-access-b6ndd\") pod \"payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:01.171967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.171933 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:01.211779 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.211733 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:01.373794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.373762 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" podUID="024635e2-f5c8-465f-8833-4488e8da9d14" containerName="payload-processing" containerID="cri-o://a837a84b1f43b0ed7e3a896172f40fe6f93db5cc2ecd1b0239b9e9738b30dd72" gracePeriod=25 Nov 01 04:11:01.469226 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.469161 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" podUID="c08d53b8-9e19-45b5-8059-8b69fbc21927" containerName="payload-pre-processing" containerID="cri-o://db2a5682f05726a361458e87ad11c8301d3801c12f7c097191707e9781ff9873" gracePeriod=25 Nov 01 04:11:01.544207 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.544178 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6"] Nov 01 04:11:01.545219 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:11:01.545179 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeccb9e1_5fd7_457b_a11f_84f8baa26120.slice/crio-a0b41c89eb5e02352cfee5a38d401ac900a2a3a4ee916545e05e655955e755a5 WatchSource:0}: Error finding container a0b41c89eb5e02352cfee5a38d401ac900a2a3a4ee916545e05e655955e755a5: Status 404 returned error can't find the container with id a0b41c89eb5e02352cfee5a38d401ac900a2a3a4ee916545e05e655955e755a5 Nov 01 04:11:01.646790 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.643656 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474"] Nov 01 04:11:01.725832 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.725797 6735 generic.go:356] "Generic (PLEG): container finished" podID="c08d53b8-9e19-45b5-8059-8b69fbc21927" containerID="db2a5682f05726a361458e87ad11c8301d3801c12f7c097191707e9781ff9873" exitCode=0 Nov 01 04:11:01.726298 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.725871 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" event={"ID":"c08d53b8-9e19-45b5-8059-8b69fbc21927","Type":"ContainerDied","Data":"db2a5682f05726a361458e87ad11c8301d3801c12f7c097191707e9781ff9873"} Nov 01 04:11:01.730099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.728720 6735 generic.go:356] "Generic (PLEG): container finished" podID="024635e2-f5c8-465f-8833-4488e8da9d14" containerID="a837a84b1f43b0ed7e3a896172f40fe6f93db5cc2ecd1b0239b9e9738b30dd72" exitCode=0 Nov 01 04:11:01.730099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.728838 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" event={"ID":"024635e2-f5c8-465f-8833-4488e8da9d14","Type":"ContainerDied","Data":"a837a84b1f43b0ed7e3a896172f40fe6f93db5cc2ecd1b0239b9e9738b30dd72"} Nov 01 04:11:01.741514 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.741484 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" event={"ID":"aeccb9e1-5fd7-457b-a11f-84f8baa26120","Type":"ContainerStarted","Data":"d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada"} Nov 01 04:11:01.741741 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.741720 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" event={"ID":"aeccb9e1-5fd7-457b-a11f-84f8baa26120","Type":"ContainerStarted","Data":"a0b41c89eb5e02352cfee5a38d401ac900a2a3a4ee916545e05e655955e755a5"} Nov 01 04:11:01.741925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.741912 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:01.743840 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.743818 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" event={"ID":"643eea33-f1c4-438b-a453-5ed690d5ca96","Type":"ContainerStarted","Data":"41f4985f7f64e739e0d110809b2a4e4ec4cb6090b5caceff65ca02cd577f26dc"} Nov 01 04:11:01.767264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.767245 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:11:01.798152 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.798094 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:11:01.859220 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.859159 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" podStartSLOduration=1.859139142 podStartE2EDuration="1.859139142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:01.855024039 +0000 UTC m=+1673.444118670" watchObservedRunningTime="2026-11-01 04:11:01.859139142 +0000 UTC m=+1673.448233772" Nov 01 04:11:01.904394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.904364 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08d53b8-9e19-45b5-8059-8b69fbc21927-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08d53b8-9e19-45b5-8059-8b69fbc21927-plugins-config-volume\") pod \"c08d53b8-9e19-45b5-8059-8b69fbc21927\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " Nov 01 04:11:01.904562 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.904476 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024635e2-f5c8-465f-8833-4488e8da9d14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024635e2-f5c8-465f-8833-4488e8da9d14-plugins-config-volume\") pod \"024635e2-f5c8-465f-8833-4488e8da9d14\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " Nov 01 04:11:01.904622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.904590 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024635e2-f5c8-465f-8833-4488e8da9d14-kube-api-access-jpdww\" (UniqueName: \"kubernetes.io/projected/024635e2-f5c8-465f-8833-4488e8da9d14-kube-api-access-jpdww\") pod \"024635e2-f5c8-465f-8833-4488e8da9d14\" (UID: \"024635e2-f5c8-465f-8833-4488e8da9d14\") " Nov 01 04:11:01.904667 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.904625 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08d53b8-9e19-45b5-8059-8b69fbc21927-kube-api-access-7gvlg\" (UniqueName: \"kubernetes.io/projected/c08d53b8-9e19-45b5-8059-8b69fbc21927-kube-api-access-7gvlg\") pod \"c08d53b8-9e19-45b5-8059-8b69fbc21927\" (UID: \"c08d53b8-9e19-45b5-8059-8b69fbc21927\") " Nov 01 04:11:01.904834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.904810 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08d53b8-9e19-45b5-8059-8b69fbc21927-plugins-config-volume" pod "c08d53b8-9e19-45b5-8059-8b69fbc21927" (UID: "c08d53b8-9e19-45b5-8059-8b69fbc21927"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:01.904921 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.904882 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024635e2-f5c8-465f-8833-4488e8da9d14-plugins-config-volume" pod "024635e2-f5c8-465f-8833-4488e8da9d14" (UID: "024635e2-f5c8-465f-8833-4488e8da9d14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:01.905548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.905528 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/024635e2-f5c8-465f-8833-4488e8da9d14-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:01.905618 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.905558 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08d53b8-9e19-45b5-8059-8b69fbc21927-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:01.906768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.906749 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024635e2-f5c8-465f-8833-4488e8da9d14-kube-api-access-jpdww" pod "024635e2-f5c8-465f-8833-4488e8da9d14" (UID: "024635e2-f5c8-465f-8833-4488e8da9d14"). InnerVolumeSpecName "kube-api-access-jpdww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:01.906910 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:01.906891 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08d53b8-9e19-45b5-8059-8b69fbc21927-kube-api-access-7gvlg" pod "c08d53b8-9e19-45b5-8059-8b69fbc21927" (UID: "c08d53b8-9e19-45b5-8059-8b69fbc21927"). InnerVolumeSpecName "kube-api-access-7gvlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:02.007186 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.006815 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpdww\" (UniqueName: \"kubernetes.io/projected/024635e2-f5c8-465f-8833-4488e8da9d14-kube-api-access-jpdww\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:02.007186 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.006853 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gvlg\" (UniqueName: \"kubernetes.io/projected/c08d53b8-9e19-45b5-8059-8b69fbc21927-kube-api-access-7gvlg\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:02.751568 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.751533 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" event={"ID":"643eea33-f1c4-438b-a453-5ed690d5ca96","Type":"ContainerStarted","Data":"1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a"} Nov 01 04:11:02.752024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.751743 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:02.753014 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.752992 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" Nov 01 04:11:02.753173 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.752995 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf" event={"ID":"c08d53b8-9e19-45b5-8059-8b69fbc21927","Type":"ContainerDied","Data":"95209985509c38c4dec7033c93b1062996f158a109dc0619b9986a00ee574d96"} Nov 01 04:11:02.753173 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.753091 6735 scope.go:122] "RemoveContainer" containerID="db2a5682f05726a361458e87ad11c8301d3801c12f7c097191707e9781ff9873" Nov 01 04:11:02.754438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.754421 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" Nov 01 04:11:02.754501 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.754458 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz" event={"ID":"024635e2-f5c8-465f-8833-4488e8da9d14","Type":"ContainerDied","Data":"327d6d7528ee9e7a3542820dca68fc9ec716ff89e6a04e6aa0009f9eb88b27af"} Nov 01 04:11:02.763895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.763876 6735 scope.go:122] "RemoveContainer" containerID="a837a84b1f43b0ed7e3a896172f40fe6f93db5cc2ecd1b0239b9e9738b30dd72" Nov 01 04:11:02.859789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.859679 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" podStartSLOduration=2.859661556 podStartE2EDuration="2.859661556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:02.859110936 +0000 UTC m=+1674.448205567" watchObservedRunningTime="2026-11-01 04:11:02.859661556 +0000 UTC m=+1674.448756191" Nov 01 04:11:02.964650 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.964614 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz"] Nov 01 04:11:02.968589 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:02.968248 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-078e0d65-d7cb67597-qmwmz"] Nov 01 04:11:03.065641 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:03.065594 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:11:03.069638 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:03.069608 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf"] Nov 01 04:11:03.073837 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:03.073789 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-078e0d65-58c4f98db7-b4sxf"] Nov 01 04:11:04.526589 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:04.526556 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024635e2-f5c8-465f-8833-4488e8da9d14" path="/var/lib/kubelet/pods/024635e2-f5c8-465f-8833-4488e8da9d14/volumes" Nov 01 04:11:04.527058 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:04.526983 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08d53b8-9e19-45b5-8059-8b69fbc21927" path="/var/lib/kubelet/pods/c08d53b8-9e19-45b5-8059-8b69fbc21927/volumes" Nov 01 04:11:13.767159 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:13.767130 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:11:13.767531 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:13.767198 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:11:18.814265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:18.814229 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb"] Nov 01 04:11:18.814861 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:18.814478 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" containerID="cri-o://f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e" gracePeriod=30 Nov 01 04:11:19.313899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.313864 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt"] Nov 01 04:11:19.314153 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.314119 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" podUID="7f000817-dda8-47bf-961b-bab5deefd69a" containerName="maas-api" containerID="cri-o://14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9" gracePeriod=30 Nov 01 04:11:19.660617 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.660594 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:11:19.767062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.766575 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f000817-dda8-47bf-961b-bab5deefd69a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f000817-dda8-47bf-961b-bab5deefd69a-openshift-service-ca\") pod \"7f000817-dda8-47bf-961b-bab5deefd69a\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " Nov 01 04:11:19.767062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.766789 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls\") pod \"7f000817-dda8-47bf-961b-bab5deefd69a\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " Nov 01 04:11:19.767062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.766851 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f000817-dda8-47bf-961b-bab5deefd69a-kube-api-access-hlbvn\" (UniqueName: \"kubernetes.io/projected/7f000817-dda8-47bf-961b-bab5deefd69a-kube-api-access-hlbvn\") pod \"7f000817-dda8-47bf-961b-bab5deefd69a\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " Nov 01 04:11:19.767062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.766933 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-metrics-certs\") pod \"7f000817-dda8-47bf-961b-bab5deefd69a\" (UID: \"7f000817-dda8-47bf-961b-bab5deefd69a\") " Nov 01 04:11:19.809081 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.807009 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls" pod "7f000817-dda8-47bf-961b-bab5deefd69a" (UID: "7f000817-dda8-47bf-961b-bab5deefd69a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:11:19.817462 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.815279 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f000817-dda8-47bf-961b-bab5deefd69a-openshift-service-ca" pod "7f000817-dda8-47bf-961b-bab5deefd69a" (UID: "7f000817-dda8-47bf-961b-bab5deefd69a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:19.822319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.822261 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f000817-dda8-47bf-961b-bab5deefd69a-kube-api-access-hlbvn" pod "7f000817-dda8-47bf-961b-bab5deefd69a" (UID: "7f000817-dda8-47bf-961b-bab5deefd69a"). InnerVolumeSpecName "kube-api-access-hlbvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:19.831342 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.829014 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-metrics-certs" pod "7f000817-dda8-47bf-961b-bab5deefd69a" (UID: "7f000817-dda8-47bf-961b-bab5deefd69a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:11:19.870116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.869998 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:19.870116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.870056 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbvn\" (UniqueName: \"kubernetes.io/projected/7f000817-dda8-47bf-961b-bab5deefd69a-kube-api-access-hlbvn\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:19.870116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.870076 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f000817-dda8-47bf-961b-bab5deefd69a-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:19.870116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.870089 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f000817-dda8-47bf-961b-bab5deefd69a-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:19.880681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.880649 6735 generic.go:356] "Generic (PLEG): container finished" podID="7f000817-dda8-47bf-961b-bab5deefd69a" containerID="14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9" exitCode=0 Nov 01 04:11:19.880821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.880705 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" Nov 01 04:11:19.880821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.880736 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" event={"ID":"7f000817-dda8-47bf-961b-bab5deefd69a","Type":"ContainerDied","Data":"14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9"} Nov 01 04:11:19.880821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.880801 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt" event={"ID":"7f000817-dda8-47bf-961b-bab5deefd69a","Type":"ContainerDied","Data":"ca48199fb818668219a72fc56c3b044db257d9a7ed49a5d284c7ea1db655e3b9"} Nov 01 04:11:19.880821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.880817 6735 scope.go:122] "RemoveContainer" containerID="14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9" Nov 01 04:11:19.898473 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.898453 6735 scope.go:122] "RemoveContainer" containerID="14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9" Nov 01 04:11:19.898878 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:19.898853 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9\": container with ID starting with 14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9 not found: ID does not exist" containerID="14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9" Nov 01 04:11:19.898953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:19.898890 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9"} err="failed to get container status \"14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9\": rpc error: code = NotFound desc = could not find container \"14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9\": container with ID starting with 14d7b58513c8ed1851745fd5b8a6e00bced60b197f4f86bed403008c92fa98c9 not found: ID does not exist" Nov 01 04:11:20.037826 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:20.037786 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt"] Nov 01 04:11:20.041796 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:20.041768 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-078e0d65-6588697649-j7dmt"] Nov 01 04:11:20.507544 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:20.507510 6735 patch_prober.go:29] interesting pod/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:11:20.507702 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:20.507581 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:11:20.528804 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:20.528769 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f000817-dda8-47bf-961b-bab5deefd69a" path="/var/lib/kubelet/pods/7f000817-dda8-47bf-961b-bab5deefd69a/volumes" Nov 01 04:11:23.521512 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:23.521481 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn"] Nov 01 04:11:23.521976 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:23.521683 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" containerID="cri-o://2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12" gracePeriod=30 Nov 01 04:11:24.089377 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.089319 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 04:11:24.230670 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.230637 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-kube-api-access-nxm8q\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-kube-api-access-nxm8q\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.230832 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.230722 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-socket\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.230832 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.230826 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-credential-socket\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.230985 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.230879 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istiod-ca-cert\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.230985 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.230972 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-podinfo\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.231133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231059 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-data\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.231133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231073 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-socket" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:24.231133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231110 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-token\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.231291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231152 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-certs\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.231291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231146 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-credential-socket" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:24.231291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231248 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-envoy\") pod \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\" (UID: \"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1\") " Nov 01 04:11:24.231567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231540 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-certs" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:24.231882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231854 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.231882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231884 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.232080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231901 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.232080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.231984 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-data" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:24.232214 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.232170 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istiod-ca-cert" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:24.233302 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.233276 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-kube-api-access-nxm8q" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "kube-api-access-nxm8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:24.233803 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.233779 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-podinfo" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:11:24.233903 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.233887 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-token" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:24.234175 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.234151 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-envoy" pod "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" (UID: "8b6b4475-8ef9-47fa-a963-e3b2d77b44b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:24.332724 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.332696 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.332724 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.332722 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.332925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.332733 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.332925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.332742 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.332925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.332751 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxm8q\" (UniqueName: \"kubernetes.io/projected/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-kube-api-access-nxm8q\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.332925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.332760 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:24.913508 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.913430 6735 generic.go:356] "Generic (PLEG): container finished" podID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerID="f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e" exitCode=0 Nov 01 04:11:24.913508 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.913472 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" Nov 01 04:11:24.913508 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.913493 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" event={"ID":"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1","Type":"ContainerDied","Data":"f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e"} Nov 01 04:11:24.913947 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.913542 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb" event={"ID":"8b6b4475-8ef9-47fa-a963-e3b2d77b44b1","Type":"ContainerDied","Data":"979156c9a5d9721b0526b291428d37b7d153fef96d0aafdc0012a380cd309a64"} Nov 01 04:11:24.913947 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.913564 6735 scope.go:122] "RemoveContainer" containerID="f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e" Nov 01 04:11:24.923167 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.923150 6735 scope.go:122] "RemoveContainer" containerID="f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e" Nov 01 04:11:24.923476 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:24.923454 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e\": container with ID starting with f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e not found: ID does not exist" containerID="f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e" Nov 01 04:11:24.923553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:24.923489 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e"} err="failed to get container status \"f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e\": rpc error: code = NotFound desc = could not find container \"f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e\": container with ID starting with f8a062003f95ccd2380baa56ae7f677e2c7bd93ad4c5751b6ea058fa22713b1e not found: ID does not exist" Nov 01 04:11:25.116441 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:25.116403 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb"] Nov 01 04:11:25.163467 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:25.163434 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-bs7hb"] Nov 01 04:11:26.528003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:26.527970 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" path="/var/lib/kubelet/pods/8b6b4475-8ef9-47fa-a963-e3b2d77b44b1/volumes" Nov 01 04:11:28.777346 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.777312 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:11:28.905962 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.905883 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-socket\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.905996 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-credential-socket\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906016 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-certs\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906078 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-envoy\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906133 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-podinfo\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906153 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-token\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-token\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906187 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istiod-ca-cert\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906188 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-socket" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:28.906415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906268 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-data\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906303 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-kube-api-access-b87jn\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-kube-api-access-b87jn\") pod \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\" (UID: \"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3\") " Nov 01 04:11:28.906415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906307 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-certs" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:28.906415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906326 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-credential-socket" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:28.906707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906618 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-data" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:28.906755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906720 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istiod-ca-cert" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:28.906978 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906934 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:28.906978 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906958 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:28.906978 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906972 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:28.907155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906987 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:28.907155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.906999 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:28.908559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.908530 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-envoy" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:28.908716 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.908692 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-token" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:28.908979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.908955 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-podinfo" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:11:28.908979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.908960 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-kube-api-access-b87jn" pod "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" (UID: "fe46460e-0ced-4bd2-9a3c-b4f9df8385d3"). InnerVolumeSpecName "kube-api-access-b87jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:28.943049 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.942997 6735 generic.go:356] "Generic (PLEG): container finished" podID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerID="2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12" exitCode=0 Nov 01 04:11:28.943206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.943057 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" Nov 01 04:11:28.943206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.943065 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" event={"ID":"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3","Type":"ContainerDied","Data":"2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12"} Nov 01 04:11:28.943206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.943121 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn" event={"ID":"fe46460e-0ced-4bd2-9a3c-b4f9df8385d3","Type":"ContainerDied","Data":"257df16a8b517660b0bc049195eca8d6f0a08ab7abf83b7cec86f3c68d5ea68f"} Nov 01 04:11:28.943206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.943145 6735 scope.go:122] "RemoveContainer" containerID="2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12" Nov 01 04:11:28.956876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.956848 6735 scope.go:122] "RemoveContainer" containerID="2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12" Nov 01 04:11:28.957235 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:28.957205 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12\": container with ID starting with 2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12 not found: ID does not exist" containerID="2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12" Nov 01 04:11:28.957340 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:28.957247 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12"} err="failed to get container status \"2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12\": rpc error: code = NotFound desc = could not find container \"2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12\": container with ID starting with 2d4b179295286868ff4eb4d12f77397b36d42b0f4c0da7eb1f84d77cf79d5e12 not found: ID does not exist" Nov 01 04:11:29.008497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.008464 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:29.008497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.008491 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:29.008667 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.008506 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:29.008667 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.008521 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87jn\" (UniqueName: \"kubernetes.io/projected/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3-kube-api-access-b87jn\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:29.098821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.098787 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn"] Nov 01 04:11:29.102983 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.102959 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-hcndn"] Nov 01 04:11:29.655516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.655479 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm"] Nov 01 04:11:29.655734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.655711 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" containerID="cri-o://421da949b99f809efc3e8813c14c2984e7d8901682d41f0d690af8f42bf92a4b" gracePeriod=30 Nov 01 04:11:29.703225 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.703190 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm"] Nov 01 04:11:29.703517 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.703486 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" containerID="cri-o://9294b869b56f12349ba67a46c1630e7eedbd66578300f40d13dd244b65e2dc8f" gracePeriod=30 Nov 01 04:11:29.706081 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.706054 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899"] Nov 01 04:11:29.706410 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.706382 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" containerID="cri-o://204a5dc4cc41966e4620ba59d75a1ceef03c06c355e16b6a33b4a27dea3349cb" gracePeriod=30 Nov 01 04:11:29.708750 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.708724 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm"] Nov 01 04:11:29.708932 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:29.708910 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" containerID="cri-o://2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4" gracePeriod=30 Nov 01 04:11:30.526896 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:30.526864 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" path="/var/lib/kubelet/pods/fe46460e-0ced-4bd2-9a3c-b4f9df8385d3/volumes" Nov 01 04:11:31.113190 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:31.113152 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-b8899 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:11:31.113542 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:31.113491 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:11:33.032618 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:33.032585 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:11:34.486970 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:34.486938 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-c5zfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:11:34.487378 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:34.487002 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:11:34.627475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:34.627435 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-prwvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:11:34.627626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:34.627489 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:11:34.880016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:34.879981 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n8gdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 01 04:11:34.880215 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:34.880056 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 01 04:11:34.980614 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:34.980584 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 04:11:35.018754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.018725 6735 generic.go:356] "Generic (PLEG): container finished" podID="0058a65d-d7aa-4969-9c60-e103cef65665" containerID="421da949b99f809efc3e8813c14c2984e7d8901682d41f0d690af8f42bf92a4b" exitCode=0 Nov 01 04:11:35.019095 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.018798 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" event={"ID":"0058a65d-d7aa-4969-9c60-e103cef65665","Type":"ContainerDied","Data":"421da949b99f809efc3e8813c14c2984e7d8901682d41f0d690af8f42bf92a4b"} Nov 01 04:11:35.020450 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.020429 6735 generic.go:356] "Generic (PLEG): container finished" podID="3aa25678-12f9-454a-890e-6ca831c89485" containerID="2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4" exitCode=0 Nov 01 04:11:35.020577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.020487 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" Nov 01 04:11:35.020577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.020548 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" event={"ID":"3aa25678-12f9-454a-890e-6ca831c89485","Type":"ContainerDied","Data":"2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4"} Nov 01 04:11:35.020694 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.020580 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm" event={"ID":"3aa25678-12f9-454a-890e-6ca831c89485","Type":"ContainerDied","Data":"d7d19f36ebaaa017e5052b86b72153f85ea1f9756f215537c6424a8b5515aa75"} Nov 01 04:11:35.020694 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.020603 6735 scope.go:122] "RemoveContainer" containerID="2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4" Nov 01 04:11:35.022937 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.022914 6735 generic.go:356] "Generic (PLEG): container finished" podID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerID="204a5dc4cc41966e4620ba59d75a1ceef03c06c355e16b6a33b4a27dea3349cb" exitCode=0 Nov 01 04:11:35.023077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.023013 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" event={"ID":"b554f3ac-a04c-40bb-b366-ea47151938ad","Type":"ContainerDied","Data":"204a5dc4cc41966e4620ba59d75a1ceef03c06c355e16b6a33b4a27dea3349cb"} Nov 01 04:11:35.025104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.024976 6735 generic.go:356] "Generic (PLEG): container finished" podID="22cb51f3-9162-4372-b68e-5092fc67600e" containerID="9294b869b56f12349ba67a46c1630e7eedbd66578300f40d13dd244b65e2dc8f" exitCode=0 Nov 01 04:11:35.025104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.025010 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" event={"ID":"22cb51f3-9162-4372-b68e-5092fc67600e","Type":"ContainerDied","Data":"9294b869b56f12349ba67a46c1630e7eedbd66578300f40d13dd244b65e2dc8f"} Nov 01 04:11:35.044298 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.044278 6735 scope.go:122] "RemoveContainer" containerID="2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4" Nov 01 04:11:35.045455 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:35.045003 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4\": container with ID starting with 2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4 not found: ID does not exist" containerID="2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4" Nov 01 04:11:35.045455 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.045422 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4"} err="failed to get container status \"2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4\": rpc error: code = NotFound desc = could not find container \"2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4\": container with ID starting with 2137752035efde9bf76cd599fe043f7023d0b8f2ebd342a95fcb1c8896c4a1c4 not found: ID does not exist" Nov 01 04:11:35.047133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.047078 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 04:11:35.073287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.073264 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 04:11:35.075269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.075254 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 04:11:35.097477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.097445 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aa25678-12f9-454a-890e-6ca831c89485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa25678-12f9-454a-890e-6ca831c89485-istio-podinfo\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.097596 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.097511 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-envoy\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.097596 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.097567 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-data\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.097729 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.097618 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa25678-12f9-454a-890e-6ca831c89485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa25678-12f9-454a-890e-6ca831c89485-istiod-ca-cert\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.097781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.097739 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-credential-socket\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.097781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.097769 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-kube-api-access-qpppp\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-kube-api-access-qpppp\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.098070 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098014 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-credential-socket" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.098172 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098095 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-istio-token\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-istio-token\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.098172 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098114 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa25678-12f9-454a-890e-6ca831c89485-istiod-ca-cert" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:35.098172 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098142 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-socket\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.098326 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098237 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-certs\") pod \"3aa25678-12f9-454a-890e-6ca831c89485\" (UID: \"3aa25678-12f9-454a-890e-6ca831c89485\") " Nov 01 04:11:35.098725 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098696 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-certs" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.098725 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098705 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-socket" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.098869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098855 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.098925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098874 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa25678-12f9-454a-890e-6ca831c89485-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.098925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098888 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.098925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.098902 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.099174 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.099152 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-data" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.101264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.101240 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-envoy" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.102275 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.102249 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa25678-12f9-454a-890e-6ca831c89485-istio-podinfo" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:11:35.102398 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.102272 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-istio-token" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.103739 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.103713 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-kube-api-access-qpppp" pod "3aa25678-12f9-454a-890e-6ca831c89485" (UID: "3aa25678-12f9-454a-890e-6ca831c89485"). InnerVolumeSpecName "kube-api-access-qpppp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.199802 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.199755 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b554f3ac-a04c-40bb-b366-ea47151938ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b554f3ac-a04c-40bb-b366-ea47151938ad-istiod-ca-cert\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.199991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.199807 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-data\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.199991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.199840 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-istio-token\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-istio-token\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.199991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.199872 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22cb51f3-9162-4372-b68e-5092fc67600e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cb51f3-9162-4372-b68e-5092fc67600e-istio-podinfo\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.199991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.199915 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-certs\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.199991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.199941 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-credential-socket\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.199991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.199981 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-credential-socket\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200023 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-envoy\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200160 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-certs\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200192 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-data" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200198 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b554f3ac-a04c-40bb-b366-ea47151938ad-istiod-ca-cert" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200208 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-certs\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200324 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-credential-socket" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200345 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-credential-socket" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200401 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-envoy\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200434 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-socket\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200465 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-data\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200494 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0058a65d-d7aa-4969-9c60-e103cef65665-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0058a65d-d7aa-4969-9c60-e103cef65665-istiod-ca-cert\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200531 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-certs" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200545 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-credential-socket\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200465 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-certs" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200598 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-socket\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200630 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-istio-token\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-istio-token\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200676 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0058a65d-d7aa-4969-9c60-e103cef65665-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0058a65d-d7aa-4969-9c60-e103cef65665-istio-podinfo\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200715 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-envoy\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200725 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-data" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.200740 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200743 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-data\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200782 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-socket" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200820 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-token\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-token\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200877 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cb51f3-9162-4372-b68e-5092fc67600e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cb51f3-9162-4372-b68e-5092fc67600e-istiod-ca-cert\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200956 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0058a65d-d7aa-4969-9c60-e103cef65665-istiod-ca-cert" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.200999 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-socket\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.201070 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-kube-api-access-2f4xr\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-kube-api-access-2f4xr\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.201156 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-podinfo\") pod \"b554f3ac-a04c-40bb-b366-ea47151938ad\" (UID: \"b554f3ac-a04c-40bb-b366-ea47151938ad\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.201253 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-kube-api-access-j76z5\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-kube-api-access-j76z5\") pod \"0058a65d-d7aa-4969-9c60-e103cef65665\" (UID: \"0058a65d-d7aa-4969-9c60-e103cef65665\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.201285 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-certs" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.201376 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-kube-api-access-p2tdq\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-kube-api-access-p2tdq\") pod \"22cb51f3-9162-4372-b68e-5092fc67600e\" (UID: \"22cb51f3-9162-4372-b68e-5092fc67600e\") " Nov 01 04:11:35.201931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.201892 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-socket" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.202534 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.202251 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-credential-socket" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.203198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203023 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-socket" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.203728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203330 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-data" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.203728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203414 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-istio-token" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.203728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203452 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-envoy" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.203728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203480 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-envoy" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.203728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203527 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-token" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203875 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203899 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203915 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa25678-12f9-454a-890e-6ca831c89485-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203928 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203942 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203956 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa25678-12f9-454a-890e-6ca831c89485-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203969 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203983 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.203997 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204013 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204026 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0058a65d-d7aa-4969-9c60-e103cef65665-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204073 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204087 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204100 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204114 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cb51f3-9162-4372-b68e-5092fc67600e-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204128 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpppp\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-kube-api-access-qpppp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204142 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204155 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa25678-12f9-454a-890e-6ca831c89485-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204168 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204183 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b554f3ac-a04c-40bb-b366-ea47151938ad-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204195 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204208 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0058a65d-d7aa-4969-9c60-e103cef65665-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.204232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.204221 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.206230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.205238 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cb51f3-9162-4372-b68e-5092fc67600e-istiod-ca-cert" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:35.206230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.205315 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22cb51f3-9162-4372-b68e-5092fc67600e-istio-podinfo" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:11:35.206230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.205606 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-podinfo" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:11:35.206230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.205754 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-kube-api-access-2f4xr" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "kube-api-access-2f4xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.206230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.205890 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-kube-api-access-p2tdq" pod "22cb51f3-9162-4372-b68e-5092fc67600e" (UID: "22cb51f3-9162-4372-b68e-5092fc67600e"). InnerVolumeSpecName "kube-api-access-p2tdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.206230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.206212 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0058a65d-d7aa-4969-9c60-e103cef65665-istio-podinfo" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:11:35.206931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.206894 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-envoy" pod "b554f3ac-a04c-40bb-b366-ea47151938ad" (UID: "b554f3ac-a04c-40bb-b366-ea47151938ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:35.207222 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.207200 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-istio-token" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.207460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.207436 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-kube-api-access-j76z5" pod "0058a65d-d7aa-4969-9c60-e103cef65665" (UID: "0058a65d-d7aa-4969-9c60-e103cef65665"). InnerVolumeSpecName "kube-api-access-j76z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305624 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0058a65d-d7aa-4969-9c60-e103cef65665-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305655 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305664 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cb51f3-9162-4372-b68e-5092fc67600e-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305674 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f4xr\" (UniqueName: \"kubernetes.io/projected/b554f3ac-a04c-40bb-b366-ea47151938ad-kube-api-access-2f4xr\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305684 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b554f3ac-a04c-40bb-b366-ea47151938ad-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305692 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76z5\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-kube-api-access-j76z5\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305702 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2tdq\" (UniqueName: \"kubernetes.io/projected/22cb51f3-9162-4372-b68e-5092fc67600e-kube-api-access-p2tdq\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.305707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305711 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0058a65d-d7aa-4969-9c60-e103cef65665-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.306154 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.305720 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cb51f3-9162-4372-b68e-5092fc67600e-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:35.596183 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596089 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz"] Nov 01 04:11:35.596857 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596836 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c08d53b8-9e19-45b5-8059-8b69fbc21927" containerName="payload-pre-processing" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596860 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08d53b8-9e19-45b5-8059-8b69fbc21927" containerName="payload-pre-processing" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596886 6735 cpu_manager.go:414] "containerMap: removing container" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596894 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596907 6735 cpu_manager.go:414] "containerMap: removing container" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596915 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596926 6735 cpu_manager.go:414] "containerMap: removing container" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596933 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596945 6735 cpu_manager.go:414] "containerMap: removing container" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" Nov 01 04:11:35.596959 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596952 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596978 6735 cpu_manager.go:414] "containerMap: removing container" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.596985 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597005 6735 cpu_manager.go:414] "containerMap: removing container" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597012 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597026 6735 cpu_manager.go:414] "containerMap: removing container" podUID="024635e2-f5c8-465f-8833-4488e8da9d14" containerName="payload-processing" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597051 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024635e2-f5c8-465f-8833-4488e8da9d14" containerName="payload-processing" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597092 6735 cpu_manager.go:414] "containerMap: removing container" podUID="7f000817-dda8-47bf-961b-bab5deefd69a" containerName="maas-api" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597099 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f000817-dda8-47bf-961b-bab5deefd69a" containerName="maas-api" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597235 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe46460e-0ced-4bd2-9a3c-b4f9df8385d3" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597249 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597261 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597273 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6b4475-8ef9-47fa-a963-e3b2d77b44b1" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597284 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa25678-12f9-454a-890e-6ca831c89485" containerName="istio-proxy" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597292 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="024635e2-f5c8-465f-8833-4488e8da9d14" containerName="payload-processing" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597306 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f000817-dda8-47bf-961b-bab5deefd69a" containerName="maas-api" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597316 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08d53b8-9e19-45b5-8059-8b69fbc21927" containerName="payload-pre-processing" Nov 01 04:11:35.597511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.597324 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" containerName="istio-proxy" Nov 01 04:11:35.602486 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.602462 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.653309 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.653276 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fb1c1f\"/\"kube-root-ca.crt\"" Nov 01 04:11:35.654179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.653300 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fb1c1f\"/\"tenantcfg-model-fb1c1f-kserve-self-signed-certs\"" Nov 01 04:11:35.654179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.653379 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fb1c1f\"/\"openshift-service-ca.crt\"" Nov 01 04:11:35.654179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.653464 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fb1c1f\"/\"default-dockercfg-4wmvl\"" Nov 01 04:11:35.656680 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.656655 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz"] Nov 01 04:11:35.674195 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.674155 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm"] Nov 01 04:11:35.680751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.680716 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-c5zfm"] Nov 01 04:11:35.710340 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.710307 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9fq\" (UniqueName: \"kubernetes.io/projected/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-kube-api-access-zg9fq\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.710522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.710362 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-dshm\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.710522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.710492 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-home\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.710619 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.710546 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tmp-dir\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.710684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.710658 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-model-cache\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.710741 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.710712 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tls-certs\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.811577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.811540 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9fq\" (UniqueName: \"kubernetes.io/projected/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-kube-api-access-zg9fq\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.811784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.811592 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-dshm\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.811784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.811647 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-home\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.811784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.811682 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tmp-dir\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.811784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.811713 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-model-cache\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.811784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.811738 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tls-certs\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.812179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.812077 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-home\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.812252 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.812232 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-model-cache\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.812420 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.812393 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tmp-dir\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.814817 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.814798 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-dshm\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.815099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.815080 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tls-certs\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.868981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.868915 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9fq\" (UniqueName: \"kubernetes.io/projected/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-kube-api-access-zg9fq\") pod \"tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:35.915464 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:35.915436 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:36.082083 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.075480 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" event={"ID":"0058a65d-d7aa-4969-9c60-e103cef65665","Type":"ContainerDied","Data":"53f790ff3609da2c8f8a1a0b2a8955cdd1dba330bed105a86a5c59ef01a62323"} Nov 01 04:11:36.082083 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.075559 6735 scope.go:122] "RemoveContainer" containerID="421da949b99f809efc3e8813c14c2984e7d8901682d41f0d690af8f42bf92a4b" Nov 01 04:11:36.082083 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.076506 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm" Nov 01 04:11:36.102758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.101586 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" event={"ID":"b554f3ac-a04c-40bb-b366-ea47151938ad","Type":"ContainerDied","Data":"3558ba7a01a6f0140a487d5756098ed19582d80ed0d5804d2026d396c8ee290e"} Nov 01 04:11:36.102758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.101738 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899" Nov 01 04:11:36.130272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.129018 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" event={"ID":"22cb51f3-9162-4372-b68e-5092fc67600e","Type":"ContainerDied","Data":"3c3b235b1c4921ff1f03dfb8ab5ea80ec1b794bc1c4651d7a664d3f8f9f82d21"} Nov 01 04:11:36.130272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.129193 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm" Nov 01 04:11:36.140116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.139573 6735 scope.go:122] "RemoveContainer" containerID="204a5dc4cc41966e4620ba59d75a1ceef03c06c355e16b6a33b4a27dea3349cb" Nov 01 04:11:36.149438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.149414 6735 scope.go:122] "RemoveContainer" containerID="9294b869b56f12349ba67a46c1630e7eedbd66578300f40d13dd244b65e2dc8f" Nov 01 04:11:36.251558 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.251524 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm"] Nov 01 04:11:36.255897 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.255871 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-prwvm"] Nov 01 04:11:36.284168 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.284144 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz"] Nov 01 04:11:36.286092 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:11:36.286066 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d3bbbc3_d7a2_4e54_bbc2_721a6abbcbd3.slice/crio-932056b931d2d47c4e89da3fd1400158e54913247e34441929c4c8e9cf74f4a4 WatchSource:0}: Error finding container 932056b931d2d47c4e89da3fd1400158e54913247e34441929c4c8e9cf74f4a4: Status 404 returned error can't find the container with id 932056b931d2d47c4e89da3fd1400158e54913247e34441929c4c8e9cf74f4a4 Nov 01 04:11:36.358981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.358955 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899"] Nov 01 04:11:36.366203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.366173 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-b8899"] Nov 01 04:11:36.467828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.467797 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm"] Nov 01 04:11:36.473474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.473441 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n8gdm"] Nov 01 04:11:36.526768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.526734 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0058a65d-d7aa-4969-9c60-e103cef65665" path="/var/lib/kubelet/pods/0058a65d-d7aa-4969-9c60-e103cef65665/volumes" Nov 01 04:11:36.527383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.527362 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cb51f3-9162-4372-b68e-5092fc67600e" path="/var/lib/kubelet/pods/22cb51f3-9162-4372-b68e-5092fc67600e/volumes" Nov 01 04:11:36.527938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.527921 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa25678-12f9-454a-890e-6ca831c89485" path="/var/lib/kubelet/pods/3aa25678-12f9-454a-890e-6ca831c89485/volumes" Nov 01 04:11:36.528918 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:36.528901 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b554f3ac-a04c-40bb-b366-ea47151938ad" path="/var/lib/kubelet/pods/b554f3ac-a04c-40bb-b366-ea47151938ad/volumes" Nov 01 04:11:37.138269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.138241 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" event={"ID":"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3","Type":"ContainerStarted","Data":"73a91d48e0be64499b5278debb74549cc035d0336ca25f8d95eb63e71598c702"} Nov 01 04:11:37.138269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.138277 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" event={"ID":"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3","Type":"ContainerStarted","Data":"932056b931d2d47c4e89da3fd1400158e54913247e34441929c4c8e9cf74f4a4"} Nov 01 04:11:37.248382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.248326 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" podStartSLOduration=2.248311253 podStartE2EDuration="2.248311253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:37.247735874 +0000 UTC m=+1708.836830497" watchObservedRunningTime="2026-11-01 04:11:37.248311253 +0000 UTC m=+1708.837405882" Nov 01 04:11:37.531555 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.531522 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5"] Nov 01 04:11:37.531734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.531714 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" podUID="a4ae00aa-d7fa-4fc0-911a-2df51505c219" containerName="maas-api" containerID="cri-o://b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf" gracePeriod=30 Nov 01 04:11:37.889821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.889797 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:11:37.916652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.916628 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:37.944897 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.944873 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls\") pod \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " Nov 01 04:11:37.945102 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.944981 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ae00aa-d7fa-4fc0-911a-2df51505c219-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae00aa-d7fa-4fc0-911a-2df51505c219-openshift-service-ca\") pod \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " Nov 01 04:11:37.945102 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.945060 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ae00aa-d7fa-4fc0-911a-2df51505c219-kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/a4ae00aa-d7fa-4fc0-911a-2df51505c219-kube-api-access-mpwnp\") pod \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " Nov 01 04:11:37.945232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.945154 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-metrics-certs\") pod \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\" (UID: \"a4ae00aa-d7fa-4fc0-911a-2df51505c219\") " Nov 01 04:11:37.945421 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.945381 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ae00aa-d7fa-4fc0-911a-2df51505c219-openshift-service-ca" pod "a4ae00aa-d7fa-4fc0-911a-2df51505c219" (UID: "a4ae00aa-d7fa-4fc0-911a-2df51505c219"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:37.945562 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.945546 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ae00aa-d7fa-4fc0-911a-2df51505c219-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:37.947327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.947302 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls" pod "a4ae00aa-d7fa-4fc0-911a-2df51505c219" (UID: "a4ae00aa-d7fa-4fc0-911a-2df51505c219"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:11:37.947401 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.947335 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ae00aa-d7fa-4fc0-911a-2df51505c219-kube-api-access-mpwnp" pod "a4ae00aa-d7fa-4fc0-911a-2df51505c219" (UID: "a4ae00aa-d7fa-4fc0-911a-2df51505c219"). InnerVolumeSpecName "kube-api-access-mpwnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:37.947401 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:37.947371 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-metrics-certs" pod "a4ae00aa-d7fa-4fc0-911a-2df51505c219" (UID: "a4ae00aa-d7fa-4fc0-911a-2df51505c219"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:11:38.048059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.047617 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/a4ae00aa-d7fa-4fc0-911a-2df51505c219-kube-api-access-mpwnp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:38.048059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.047646 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:38.048059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.047662 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ae00aa-d7fa-4fc0-911a-2df51505c219-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:38.146801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.146715 6735 generic.go:356] "Generic (PLEG): container finished" podID="a4ae00aa-d7fa-4fc0-911a-2df51505c219" containerID="b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf" exitCode=0 Nov 01 04:11:38.146801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.146775 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" Nov 01 04:11:38.147269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.146808 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" event={"ID":"a4ae00aa-d7fa-4fc0-911a-2df51505c219","Type":"ContainerDied","Data":"b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf"} Nov 01 04:11:38.147269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.146890 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5" event={"ID":"a4ae00aa-d7fa-4fc0-911a-2df51505c219","Type":"ContainerDied","Data":"b33a8b569ccd4f50d61fa8f26d48fcc4e44c075c580aa064c2856e30b1cf60a8"} Nov 01 04:11:38.147269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.146913 6735 scope.go:122] "RemoveContainer" containerID="b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf" Nov 01 04:11:38.157864 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.157840 6735 scope.go:122] "RemoveContainer" containerID="b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf" Nov 01 04:11:38.158226 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:38.158202 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf\": container with ID starting with b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf not found: ID does not exist" containerID="b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf" Nov 01 04:11:38.158295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.158236 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf"} err="failed to get container status \"b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf\": rpc error: code = NotFound desc = could not find container \"b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf\": container with ID starting with b55b85017d2578252d4127b4d7892afc051562a9bd6bbdc25a13845e69c19fdf not found: ID does not exist" Nov 01 04:11:38.313426 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.313383 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5"] Nov 01 04:11:38.320970 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.320940 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6dd36b-66cd9bb4b5-s28t5"] Nov 01 04:11:38.527987 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:38.527952 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ae00aa-d7fa-4fc0-911a-2df51505c219" path="/var/lib/kubelet/pods/a4ae00aa-d7fa-4fc0-911a-2df51505c219/volumes" Nov 01 04:11:40.959608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:40.959572 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv"] Nov 01 04:11:40.959990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:40.959794 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" containerID="cri-o://f07a8cc15d0e3649a917efbc92a2762fc68ddc9cab857e8cab63d9753fbe6719" gracePeriod=30 Nov 01 04:11:44.802800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.802763 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg"] Nov 01 04:11:44.803503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.803473 6735 cpu_manager.go:414] "containerMap: removing container" podUID="a4ae00aa-d7fa-4fc0-911a-2df51505c219" containerName="maas-api" Nov 01 04:11:44.803503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.803495 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ae00aa-d7fa-4fc0-911a-2df51505c219" containerName="maas-api" Nov 01 04:11:44.803650 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.803598 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ae00aa-d7fa-4fc0-911a-2df51505c219" containerName="maas-api" Nov 01 04:11:44.808526 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.808508 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:44.857495 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.857464 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-157fed-serving-cert\"" Nov 01 04:11:44.908492 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.908438 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg"] Nov 01 04:11:44.950111 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.950072 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdfc\" (UniqueName: \"kubernetes.io/projected/e59f4b35-8965-419a-a5e1-938ee6b4f31f-kube-api-access-dzdfc\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:44.950287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.950213 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:44.950388 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.950367 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-metrics-certs\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:44.950466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:44.950452 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f4b35-8965-419a-a5e1-938ee6b4f31f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.051438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.051408 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdfc\" (UniqueName: \"kubernetes.io/projected/e59f4b35-8965-419a-a5e1-938ee6b4f31f-kube-api-access-dzdfc\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.051608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.051491 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.051608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.051568 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-metrics-certs\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.051749 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.051609 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f4b35-8965-419a-a5e1-938ee6b4f31f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.051749 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:45.051619 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-serving-cert: secret "maas-api-e2e-ipp-a-157fed-serving-cert" not found Nov 01 04:11:45.051749 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:45.051691 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls podName:e59f4b35-8965-419a-a5e1-938ee6b4f31f nodeName:}" failed. No retries permitted until 2026-11-01 04:11:45.551676069 +0000 UTC m=+1717.140770677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls") pod "maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" (UID: "e59f4b35-8965-419a-a5e1-938ee6b4f31f") : secret "maas-api-e2e-ipp-a-157fed-serving-cert" not found Nov 01 04:11:45.052216 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.052198 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f4b35-8965-419a-a5e1-938ee6b4f31f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.054135 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.054084 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-metrics-certs\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.113258 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.113172 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdfc\" (UniqueName: \"kubernetes.io/projected/e59f4b35-8965-419a-a5e1-938ee6b4f31f-kube-api-access-dzdfc\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.557138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.557104 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.559849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.559826 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.722758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.722730 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:45.916025 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.915932 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:45.929013 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:45.928990 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:46.094922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.094892 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg"] Nov 01 04:11:46.096400 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:11:46.096377 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59f4b35_8965_419a_a5e1_938ee6b4f31f.slice/crio-a7a43692681b4d99936afc61d160c04b0c8d782b4a1bf2f5e008157681203b26 WatchSource:0}: Error finding container a7a43692681b4d99936afc61d160c04b0c8d782b4a1bf2f5e008157681203b26: Status 404 returned error can't find the container with id a7a43692681b4d99936afc61d160c04b0c8d782b4a1bf2f5e008157681203b26 Nov 01 04:11:46.098273 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.098253 6735 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 04:11:46.203945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.203864 6735 generic.go:356] "Generic (PLEG): container finished" podID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerID="f07a8cc15d0e3649a917efbc92a2762fc68ddc9cab857e8cab63d9753fbe6719" exitCode=0 Nov 01 04:11:46.204126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.203935 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" event={"ID":"1380b1a8-c7ec-448b-ba69-4e4321c073a7","Type":"ContainerDied","Data":"f07a8cc15d0e3649a917efbc92a2762fc68ddc9cab857e8cab63d9753fbe6719"} Nov 01 04:11:46.204126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.203992 6735 scope.go:122] "RemoveContainer" containerID="ef1e1d37adfe2e4019f3a3696231588e2088b3275988d50113c02607bc52a9a2" Nov 01 04:11:46.205369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.205348 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" event={"ID":"e59f4b35-8965-419a-a5e1-938ee6b4f31f","Type":"ContainerStarted","Data":"a7a43692681b4d99936afc61d160c04b0c8d782b4a1bf2f5e008157681203b26"} Nov 01 04:11:46.218644 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.218626 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:11:46.245470 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.245452 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:11:46.367367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367337 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-certs\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.367533 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367386 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-credential-socket\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.367533 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367433 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-data\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.367533 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367501 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-socket\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.367709 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367535 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-kube-api-access-4jbrv\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-kube-api-access-4jbrv\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.367709 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367615 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-podinfo\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.367709 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367613 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-certs" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:46.367709 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367699 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-token\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-token\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.367915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367726 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-data" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:46.367915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367744 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-credential-socket" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:46.367915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367765 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-socket" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:46.367915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367767 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istiod-ca-cert\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.368153 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.367932 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-envoy\") pod \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\" (UID: \"1380b1a8-c7ec-448b-ba69-4e4321c073a7\") " Nov 01 04:11:46.368208 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.368171 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istiod-ca-cert" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:46.368727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.368677 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.368727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.368702 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.368727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.368719 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.368966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.368732 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.368966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.368746 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.370199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.370171 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-podinfo" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:11:46.370282 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.370247 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-token" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:46.370367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.370313 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-envoy" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:11:46.370713 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.370692 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-kube-api-access-4jbrv" pod "1380b1a8-c7ec-448b-ba69-4e4321c073a7" (UID: "1380b1a8-c7ec-448b-ba69-4e4321c073a7"). InnerVolumeSpecName "kube-api-access-4jbrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:46.470551 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.470516 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbrv\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-kube-api-access-4jbrv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.470551 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.470555 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.470805 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.470570 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.470805 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.470583 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1380b1a8-c7ec-448b-ba69-4e4321c073a7-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:46.946642 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:46.946551 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:11:47.223976 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:47.223943 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" Nov 01 04:11:47.223976 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:47.223940 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv" event={"ID":"1380b1a8-c7ec-448b-ba69-4e4321c073a7","Type":"ContainerDied","Data":"3bdaeb63f4e31b16589d1bc49f737495ac246e2f7b7de4563cab74984eecbcd4"} Nov 01 04:11:47.224226 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:47.224101 6735 scope.go:122] "RemoveContainer" containerID="f07a8cc15d0e3649a917efbc92a2762fc68ddc9cab857e8cab63d9753fbe6719" Nov 01 04:11:47.419661 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:47.419624 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv"] Nov 01 04:11:47.423831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:47.423740 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-78dad9-openshift-default-b98fc5575-m4nsv"] Nov 01 04:11:48.233428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.233390 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" event={"ID":"e59f4b35-8965-419a-a5e1-938ee6b4f31f","Type":"ContainerStarted","Data":"170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2"} Nov 01 04:11:48.233833 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.233457 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:48.348452 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.348401 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" podStartSLOduration=3.3384865120000002 podStartE2EDuration="4.348387447s" totalImagesPullingTime="1.009900935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:48.34691243 +0000 UTC m=+1719.936007060" watchObservedRunningTime="2026-11-01 04:11:48.348387447 +0000 UTC m=+1719.937482076" Nov 01 04:11:48.526408 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.526326 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" path="/var/lib/kubelet/pods/1380b1a8-c7ec-448b-ba69-4e4321c073a7/volumes" Nov 01 04:11:48.669319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.669289 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl"] Nov 01 04:11:48.670150 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.670125 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" Nov 01 04:11:48.670150 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.670147 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" Nov 01 04:11:48.670348 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.670163 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" Nov 01 04:11:48.670348 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.670169 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" Nov 01 04:11:48.670451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.670345 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" Nov 01 04:11:48.670451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.670361 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1380b1a8-c7ec-448b-ba69-4e4321c073a7" containerName="istio-proxy" Nov 01 04:11:48.675237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.675216 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.724498 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.724463 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5b71cd-openshift-default-dockercfg-hchtf\"" Nov 01 04:11:48.740970 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.740936 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl"] Nov 01 04:11:48.814585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814504 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-credential-socket\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814574 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-socket\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814756 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814647 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-envoy\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814756 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814672 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-certs\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814749 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-podinfo\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814801 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-token\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814900 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814873 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4nd\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-kube-api-access-qf4nd\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814908 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-data\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.814999 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.814944 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istiod-ca-cert\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.921859 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-credential-socket\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.921925 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-socket\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.921944 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-envoy\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.921962 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-certs\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.922004 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-podinfo\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.922085 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-token\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.922148 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4nd\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-kube-api-access-qf4nd\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.922183 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-data\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.922224 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istiod-ca-cert\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.922899 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istiod-ca-cert\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.923112 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-credential-socket\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.923258 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-socket\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.924006 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-certs\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.932060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.924846 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-data\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.968458 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.962639 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-podinfo\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.968458 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.964311 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-envoy\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.984330 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.984260 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4nd\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-kube-api-access-qf4nd\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:48.993097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:48.993006 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-token\") pod \"e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:49.288418 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:49.288388 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:49.730631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:49.730600 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl"] Nov 01 04:11:49.732214 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:11:49.732190 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fed2dc4_159e_4e21_a2ac_ada2f5b943dc.slice/crio-b4ce2995f1298cb23a77f7fc3613e084192b644406497497929b7a02f9cd019d WatchSource:0}: Error finding container b4ce2995f1298cb23a77f7fc3613e084192b644406497497929b7a02f9cd019d: Status 404 returned error can't find the container with id b4ce2995f1298cb23a77f7fc3613e084192b644406497497929b7a02f9cd019d Nov 01 04:11:49.734734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:49.734696 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:11:49.734844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:49.734778 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:11:49.734844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:49.734825 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:11:50.248024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:50.247988 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" event={"ID":"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc","Type":"ContainerStarted","Data":"bbdb520cdda18265b0f29918040950a03f584822d0b8316d293b21f865b82e40"} Nov 01 04:11:50.248204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:50.248066 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" event={"ID":"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc","Type":"ContainerStarted","Data":"b4ce2995f1298cb23a77f7fc3613e084192b644406497497929b7a02f9cd019d"} Nov 01 04:11:50.289268 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:50.289237 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:50.295422 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:50.295398 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:50.424908 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:50.424845 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podStartSLOduration=2.424821775 podStartE2EDuration="2.424821775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:50.412987763 +0000 UTC m=+1722.002082395" watchObservedRunningTime="2026-11-01 04:11:50.424821775 +0000 UTC m=+1722.013916404" Nov 01 04:11:51.255509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:51.255470 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:51.256458 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:51.256437 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:11:53.302393 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:53.302361 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb"] Nov 01 04:11:53.408044 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:53.408002 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf"] Nov 01 04:11:54.245993 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:54.245964 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:11:57.328171 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.328141 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7"] Nov 01 04:11:57.332291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.332270 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.383703 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.383658 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5b71cd\"" Nov 01 04:11:57.386872 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.386837 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7"] Nov 01 04:11:57.416060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.408224 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdnj\" (UniqueName: \"kubernetes.io/projected/13819afa-cd9e-4f16-b204-ad15d67a8d5d-kube-api-access-xtdnj\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.416060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.408405 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13819afa-cd9e-4f16-b204-ad15d67a8d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.431583 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.430950 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5b71cd-dockercfg-sxwh7\"" Nov 01 04:11:57.499762 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.499513 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk"] Nov 01 04:11:57.505721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.505693 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:57.510302 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.510259 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdnj\" (UniqueName: \"kubernetes.io/projected/13819afa-cd9e-4f16-b204-ad15d67a8d5d-kube-api-access-xtdnj\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.510744 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.510720 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13819afa-cd9e-4f16-b204-ad15d67a8d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.511580 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.511560 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13819afa-cd9e-4f16-b204-ad15d67a8d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.548257 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.548223 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk"] Nov 01 04:11:57.567828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.567797 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdnj\" (UniqueName: \"kubernetes.io/projected/13819afa-cd9e-4f16-b204-ad15d67a8d5d-kube-api-access-xtdnj\") pod \"payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.612457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.612366 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfh47\" (UniqueName: \"kubernetes.io/projected/eb9ad832-5e17-4b89-b8f5-62665639f4ed-kube-api-access-pfh47\") pod \"payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:57.612767 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.612735 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb9ad832-5e17-4b89-b8f5-62665639f4ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:57.646093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.646058 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:57.715017 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.714970 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfh47\" (UniqueName: \"kubernetes.io/projected/eb9ad832-5e17-4b89-b8f5-62665639f4ed-kube-api-access-pfh47\") pod \"payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:57.715231 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.715203 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb9ad832-5e17-4b89-b8f5-62665639f4ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:57.716813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.716784 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb9ad832-5e17-4b89-b8f5-62665639f4ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:57.773450 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.773423 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfh47\" (UniqueName: \"kubernetes.io/projected/eb9ad832-5e17-4b89-b8f5-62665639f4ed-kube-api-access-pfh47\") pod \"payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:57.830194 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:57.830160 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:58.016007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.015973 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7"] Nov 01 04:11:58.016906 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:11:58.016879 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13819afa_cd9e_4f16_b204_ad15d67a8d5d.slice/crio-2173eb1219b993ca85cb53a1c9b5cccce9c5533fb35cbee8a370d691500337c8 WatchSource:0}: Error finding container 2173eb1219b993ca85cb53a1c9b5cccce9c5533fb35cbee8a370d691500337c8: Status 404 returned error can't find the container with id 2173eb1219b993ca85cb53a1c9b5cccce9c5533fb35cbee8a370d691500337c8 Nov 01 04:11:58.203115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.200866 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk"] Nov 01 04:11:58.303153 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.303096 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" podUID="20da01e7-4942-4ced-a858-687b8d77ae18" containerName="payload-processing" containerID="cri-o://dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de" gracePeriod=25 Nov 01 04:11:58.305279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.305250 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" event={"ID":"eb9ad832-5e17-4b89-b8f5-62665639f4ed","Type":"ContainerStarted","Data":"dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d"} Nov 01 04:11:58.305427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.305303 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" event={"ID":"eb9ad832-5e17-4b89-b8f5-62665639f4ed","Type":"ContainerStarted","Data":"5827031eab872afc43bbb1673fb56814d9c6717f1f9368cf126cb302dfa9325e"} Nov 01 04:11:58.305427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.305353 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:11:58.307104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.307070 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" event={"ID":"13819afa-cd9e-4f16-b204-ad15d67a8d5d","Type":"ContainerStarted","Data":"55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53"} Nov 01 04:11:58.307227 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.307120 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" event={"ID":"13819afa-cd9e-4f16-b204-ad15d67a8d5d","Type":"ContainerStarted","Data":"2173eb1219b993ca85cb53a1c9b5cccce9c5533fb35cbee8a370d691500337c8"} Nov 01 04:11:58.307227 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.307143 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:11:58.408744 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.408705 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" podUID="1f914c94-2cd3-4576-854b-6b0764e120d6" containerName="payload-pre-processing" containerID="cri-o://8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362" gracePeriod=25 Nov 01 04:11:58.418148 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.418103 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" podStartSLOduration=1.418089392 podStartE2EDuration="1.418089392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:58.414813733 +0000 UTC m=+1730.003908364" watchObservedRunningTime="2026-11-01 04:11:58.418089392 +0000 UTC m=+1730.007184020" Nov 01 04:11:58.544088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.541412 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" podStartSLOduration=1.541391096 podStartE2EDuration="1.541391096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:58.524257297 +0000 UTC m=+1730.113351927" watchObservedRunningTime="2026-11-01 04:11:58.541391096 +0000 UTC m=+1730.130485728" Nov 01 04:11:58.688544 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.688128 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:11:58.738732 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.738648 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20da01e7-4942-4ced-a858-687b8d77ae18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da01e7-4942-4ced-a858-687b8d77ae18-plugins-config-volume\") pod \"20da01e7-4942-4ced-a858-687b8d77ae18\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " Nov 01 04:11:58.738873 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.738758 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20da01e7-4942-4ced-a858-687b8d77ae18-kube-api-access-vzg8s\" (UniqueName: \"kubernetes.io/projected/20da01e7-4942-4ced-a858-687b8d77ae18-kube-api-access-vzg8s\") pod \"20da01e7-4942-4ced-a858-687b8d77ae18\" (UID: \"20da01e7-4942-4ced-a858-687b8d77ae18\") " Nov 01 04:11:58.739191 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.739152 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20da01e7-4942-4ced-a858-687b8d77ae18-plugins-config-volume" pod "20da01e7-4942-4ced-a858-687b8d77ae18" (UID: "20da01e7-4942-4ced-a858-687b8d77ae18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:58.739561 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.739534 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20da01e7-4942-4ced-a858-687b8d77ae18-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:58.741996 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.741966 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20da01e7-4942-4ced-a858-687b8d77ae18-kube-api-access-vzg8s" pod "20da01e7-4942-4ced-a858-687b8d77ae18" (UID: "20da01e7-4942-4ced-a858-687b8d77ae18"). InnerVolumeSpecName "kube-api-access-vzg8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:58.809761 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.809733 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:11:58.840506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.840471 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzg8s\" (UniqueName: \"kubernetes.io/projected/20da01e7-4942-4ced-a858-687b8d77ae18-kube-api-access-vzg8s\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:58.941462 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.941434 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f914c94-2cd3-4576-854b-6b0764e120d6-kube-api-access-lgdbb\" (UniqueName: \"kubernetes.io/projected/1f914c94-2cd3-4576-854b-6b0764e120d6-kube-api-access-lgdbb\") pod \"1f914c94-2cd3-4576-854b-6b0764e120d6\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " Nov 01 04:11:58.941632 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.941521 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f914c94-2cd3-4576-854b-6b0764e120d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f914c94-2cd3-4576-854b-6b0764e120d6-plugins-config-volume\") pod \"1f914c94-2cd3-4576-854b-6b0764e120d6\" (UID: \"1f914c94-2cd3-4576-854b-6b0764e120d6\") " Nov 01 04:11:58.941941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.941913 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f914c94-2cd3-4576-854b-6b0764e120d6-plugins-config-volume" pod "1f914c94-2cd3-4576-854b-6b0764e120d6" (UID: "1f914c94-2cd3-4576-854b-6b0764e120d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:11:58.942060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.942024 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f914c94-2cd3-4576-854b-6b0764e120d6-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:58.943817 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:58.943786 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f914c94-2cd3-4576-854b-6b0764e120d6-kube-api-access-lgdbb" pod "1f914c94-2cd3-4576-854b-6b0764e120d6" (UID: "1f914c94-2cd3-4576-854b-6b0764e120d6"). InnerVolumeSpecName "kube-api-access-lgdbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:11:59.043764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.043680 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgdbb\" (UniqueName: \"kubernetes.io/projected/1f914c94-2cd3-4576-854b-6b0764e120d6-kube-api-access-lgdbb\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:11:59.314149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.314065 6735 generic.go:356] "Generic (PLEG): container finished" podID="20da01e7-4942-4ced-a858-687b8d77ae18" containerID="dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de" exitCode=0 Nov 01 04:11:59.314321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.314175 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" Nov 01 04:11:59.314321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.314200 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" event={"ID":"20da01e7-4942-4ced-a858-687b8d77ae18","Type":"ContainerDied","Data":"dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de"} Nov 01 04:11:59.314321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.314248 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb" event={"ID":"20da01e7-4942-4ced-a858-687b8d77ae18","Type":"ContainerDied","Data":"3d03c273368cd76ca9b3082f7615217b6424c9f7e08de0823378a1484f52764c"} Nov 01 04:11:59.314321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.314274 6735 scope.go:122] "RemoveContainer" containerID="dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de" Nov 01 04:11:59.315640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.315615 6735 generic.go:356] "Generic (PLEG): container finished" podID="1f914c94-2cd3-4576-854b-6b0764e120d6" containerID="8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362" exitCode=0 Nov 01 04:11:59.315754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.315653 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" Nov 01 04:11:59.315754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.315688 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" event={"ID":"1f914c94-2cd3-4576-854b-6b0764e120d6","Type":"ContainerDied","Data":"8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362"} Nov 01 04:11:59.315754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.315730 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf" event={"ID":"1f914c94-2cd3-4576-854b-6b0764e120d6","Type":"ContainerDied","Data":"2a6c43473f23bbb949d721578552bf3ee115b14423359dfad9e350dd15055aba"} Nov 01 04:11:59.325160 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.325139 6735 scope.go:122] "RemoveContainer" containerID="dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de" Nov 01 04:11:59.325459 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:59.325441 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de\": container with ID starting with dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de not found: ID does not exist" containerID="dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de" Nov 01 04:11:59.325517 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.325465 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de"} err="failed to get container status \"dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de\": rpc error: code = NotFound desc = could not find container \"dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de\": container with ID starting with dfc44c227a383e8b679523b59a028b4dd96b8391b80ea52d905cdd1a7e7212de not found: ID does not exist" Nov 01 04:11:59.325517 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.325483 6735 scope.go:122] "RemoveContainer" containerID="8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362" Nov 01 04:11:59.335068 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.335048 6735 scope.go:122] "RemoveContainer" containerID="8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362" Nov 01 04:11:59.335329 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:11:59.335311 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362\": container with ID starting with 8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362 not found: ID does not exist" containerID="8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362" Nov 01 04:11:59.335373 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.335337 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362"} err="failed to get container status \"8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362\": rpc error: code = NotFound desc = could not find container \"8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362\": container with ID starting with 8c8181c65b2797841f27fda01b5dfe6132eb651754422efb417eb7014efab362 not found: ID does not exist" Nov 01 04:11:59.469224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.469188 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb"] Nov 01 04:11:59.473248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.473223 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dc980b-7cd65fc645-qp2nb"] Nov 01 04:11:59.576303 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.576091 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf"] Nov 01 04:11:59.585606 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:11:59.585581 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dc980b-74d76b8695-896gf"] Nov 01 04:12:00.527357 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:00.527319 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f914c94-2cd3-4576-854b-6b0764e120d6" path="/var/lib/kubelet/pods/1f914c94-2cd3-4576-854b-6b0764e120d6/volumes" Nov 01 04:12:00.527897 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:00.527881 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20da01e7-4942-4ced-a858-687b8d77ae18" path="/var/lib/kubelet/pods/20da01e7-4942-4ced-a858-687b8d77ae18/volumes" Nov 01 04:12:03.001202 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:03.001164 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:12:09.318879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:09.318848 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:12:09.319420 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:09.318924 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:12:09.369500 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:09.369469 6735 scope.go:122] "RemoveContainer" containerID="c244467a0f52da67d45a2b96074c0c96bcd901f98d4d5d5764c36bc12f092bd3" Nov 01 04:12:10.820091 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:10.820058 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj"] Nov 01 04:12:10.820492 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:10.820245 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" containerID="cri-o://5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc" gracePeriod=30 Nov 01 04:12:10.830400 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:10.830374 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg"] Nov 01 04:12:10.830640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:10.830558 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" containerID="cri-o://e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1" gracePeriod=30 Nov 01 04:12:10.837676 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:10.837649 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk"] Nov 01 04:12:10.837872 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:10.837838 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" containerID="cri-o://38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249" gracePeriod=30 Nov 01 04:12:16.114302 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.114273 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:12:16.151753 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.151729 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:12:16.155163 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.155145 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:12:16.275748 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275705 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-socket\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.275934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275762 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-envoy\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.275934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275798 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-token\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-token\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.275934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275851 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-credential-socket\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.275934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275879 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-credential-socket\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.275934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275906 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-token\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-token\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275963 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-data\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.275994 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-socket" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276008 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-certs\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276074 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-istio-token\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-istio-token\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276108 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-certs\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276141 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-certs\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276169 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-podinfo\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276210 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-data\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276279 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-socket\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.276328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276316 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-envoy\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276352 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-credential-socket\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276379 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-envoy\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276402 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-podinfo\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276458 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902b46b1-e745-435e-bbfa-94735bee93bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902b46b1-e745-435e-bbfa-94735bee93bf-istiod-ca-cert\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276571 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c210ff5-234b-495f-835e-768b3f5a5b88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c210ff5-234b-495f-835e-768b3f5a5b88-istiod-ca-cert\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276576 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-certs" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276600 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-data\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276629 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a37e982-6885-4ad9-a0f9-7604756f6612-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a37e982-6885-4ad9-a0f9-7604756f6612-istiod-ca-cert\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276674 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-kube-api-access-sz4zf\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-kube-api-access-sz4zf\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276703 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-kube-api-access-lgs44\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-kube-api-access-lgs44\") pod \"1a37e982-6885-4ad9-a0f9-7604756f6612\" (UID: \"1a37e982-6885-4ad9-a0f9-7604756f6612\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276746 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/902b46b1-e745-435e-bbfa-94735bee93bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902b46b1-e745-435e-bbfa-94735bee93bf-istio-podinfo\") pod \"902b46b1-e745-435e-bbfa-94735bee93bf\" (UID: \"902b46b1-e745-435e-bbfa-94735bee93bf\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276786 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-socket\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276807 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-credential-socket" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.276825 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-kube-api-access-pq9ps\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-kube-api-access-pq9ps\") pod \"3c210ff5-234b-495f-835e-768b3f5a5b88\" (UID: \"3c210ff5-234b-495f-835e-768b3f5a5b88\") " Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.277003 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-credential-socket" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.277550 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.277575 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.277590 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.277624 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.277636 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-certs" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.279683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.278582 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-data" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.280795 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.278885 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-certs" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.280795 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.280011 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-socket" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281018 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-data" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281248 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-token" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281465 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-envoy" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281478 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-data" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281508 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-kube-api-access-pq9ps" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "kube-api-access-pq9ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281534 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-token" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281557 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-envoy" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281829 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a37e982-6885-4ad9-a0f9-7604756f6612-istiod-ca-cert" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:12:16.281869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.281853 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-envoy" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.282430 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.282401 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-istio-token" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:16.283166 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.283142 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-kube-api-access-lgs44" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "kube-api-access-lgs44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:16.283258 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.283203 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-credential-socket" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.283394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.283377 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/902b46b1-e745-435e-bbfa-94735bee93bf-istio-podinfo" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:12:16.283507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.283485 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c210ff5-234b-495f-835e-768b3f5a5b88-istiod-ca-cert" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:12:16.283733 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.283711 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-socket" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:12:16.283946 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.283927 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902b46b1-e745-435e-bbfa-94735bee93bf-istiod-ca-cert" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:12:16.284564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.284541 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-podinfo" pod "1a37e982-6885-4ad9-a0f9-7604756f6612" (UID: "1a37e982-6885-4ad9-a0f9-7604756f6612"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:12:16.285382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.285363 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-podinfo" pod "3c210ff5-234b-495f-835e-768b3f5a5b88" (UID: "3c210ff5-234b-495f-835e-768b3f5a5b88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:12:16.285601 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.285583 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-kube-api-access-sz4zf" pod "902b46b1-e745-435e-bbfa-94735bee93bf" (UID: "902b46b1-e745-435e-bbfa-94735bee93bf"). InnerVolumeSpecName "kube-api-access-sz4zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378850 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz4zf\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-kube-api-access-sz4zf\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378876 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgs44\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-kube-api-access-lgs44\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378886 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902b46b1-e745-435e-bbfa-94735bee93bf-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378897 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378907 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq9ps\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-kube-api-access-pq9ps\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378915 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378924 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.378931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378933 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378941 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378949 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378956 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902b46b1-e745-435e-bbfa-94735bee93bf-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378966 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378974 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378982 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378991 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.378998 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.379006 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902b46b1-e745-435e-bbfa-94735bee93bf-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.379013 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.379021 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c210ff5-234b-495f-835e-768b3f5a5b88-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.379029 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902b46b1-e745-435e-bbfa-94735bee93bf-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.379062 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c210ff5-234b-495f-835e-768b3f5a5b88-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.379069 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a37e982-6885-4ad9-a0f9-7604756f6612-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.379311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.379077 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a37e982-6885-4ad9-a0f9-7604756f6612-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:16.510923 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.510886 6735 generic.go:356] "Generic (PLEG): container finished" podID="902b46b1-e745-435e-bbfa-94735bee93bf" containerID="e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1" exitCode=0 Nov 01 04:12:16.511142 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.510953 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" Nov 01 04:12:16.511142 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.510980 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" event={"ID":"902b46b1-e745-435e-bbfa-94735bee93bf","Type":"ContainerDied","Data":"e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1"} Nov 01 04:12:16.511142 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.511085 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg" event={"ID":"902b46b1-e745-435e-bbfa-94735bee93bf","Type":"ContainerDied","Data":"0831679dbfdc194cedfc397459efa6178c28cd7d715992ea9ad715c210d7293e"} Nov 01 04:12:16.511142 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.511116 6735 scope.go:122] "RemoveContainer" containerID="e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1" Nov 01 04:12:16.512463 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.512441 6735 generic.go:356] "Generic (PLEG): container finished" podID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerID="5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc" exitCode=0 Nov 01 04:12:16.512840 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.512568 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" event={"ID":"3c210ff5-234b-495f-835e-768b3f5a5b88","Type":"ContainerDied","Data":"5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc"} Nov 01 04:12:16.512840 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.512582 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" Nov 01 04:12:16.512840 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.512610 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj" event={"ID":"3c210ff5-234b-495f-835e-768b3f5a5b88","Type":"ContainerDied","Data":"373548ce25fb204eea6643dd0489a3e2b66c238e336d957f17af88d4f06bad9c"} Nov 01 04:12:16.526061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.525779 6735 generic.go:356] "Generic (PLEG): container finished" podID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerID="38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249" exitCode=0 Nov 01 04:12:16.526061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.525991 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" Nov 01 04:12:16.526371 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.526352 6735 scope.go:122] "RemoveContainer" containerID="e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1" Nov 01 04:12:16.526796 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:16.526774 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1\": container with ID starting with e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1 not found: ID does not exist" containerID="e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1" Nov 01 04:12:16.526941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.526916 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1"} err="failed to get container status \"e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1\": rpc error: code = NotFound desc = could not find container \"e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1\": container with ID starting with e8825b02e18c48d370d755686966518eceff449cada9c29b51a7f755cfc164e1 not found: ID does not exist" Nov 01 04:12:16.527641 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.527624 6735 scope.go:122] "RemoveContainer" containerID="5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc" Nov 01 04:12:16.534678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.534650 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" event={"ID":"1a37e982-6885-4ad9-a0f9-7604756f6612","Type":"ContainerDied","Data":"38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249"} Nov 01 04:12:16.534759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.534690 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk" event={"ID":"1a37e982-6885-4ad9-a0f9-7604756f6612","Type":"ContainerDied","Data":"d2119309245bbeb546e54b6ffcd2cabe5c426ea6a1f5d2fadc507a37d637e5c2"} Nov 01 04:12:16.542552 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.542530 6735 scope.go:122] "RemoveContainer" containerID="5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc" Nov 01 04:12:16.542873 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:16.542854 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc\": container with ID starting with 5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc not found: ID does not exist" containerID="5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc" Nov 01 04:12:16.542982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.542878 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc"} err="failed to get container status \"5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc\": rpc error: code = NotFound desc = could not find container \"5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc\": container with ID starting with 5fd3696e25d483053091df9fea3763fb104bd0da2d900d9b3a4fc89e4f2c07fc not found: ID does not exist" Nov 01 04:12:16.542982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.542896 6735 scope.go:122] "RemoveContainer" containerID="38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249" Nov 01 04:12:16.554577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.554478 6735 scope.go:122] "RemoveContainer" containerID="38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249" Nov 01 04:12:16.554832 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:16.554806 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249\": container with ID starting with 38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249 not found: ID does not exist" containerID="38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249" Nov 01 04:12:16.554942 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.554843 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249"} err="failed to get container status \"38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249\": rpc error: code = NotFound desc = could not find container \"38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249\": container with ID starting with 38ddb904c92f668691c1826b2512ab5c939e32bec9224c6f773e1a05c545a249 not found: ID does not exist" Nov 01 04:12:16.678402 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.678315 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg"] Nov 01 04:12:16.681838 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.681809 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-whgbg"] Nov 01 04:12:16.784282 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.784254 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk"] Nov 01 04:12:16.788267 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.788241 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-k49wk"] Nov 01 04:12:16.892894 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.892869 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj"] Nov 01 04:12:16.899098 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:16.899057 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-078e0d65-openshift-default-66569d64b9-mk9lj"] Nov 01 04:12:18.301405 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:18.301371 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:12:18.527105 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:18.527072 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" path="/var/lib/kubelet/pods/1a37e982-6885-4ad9-a0f9-7604756f6612/volumes" Nov 01 04:12:18.527567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:18.527554 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" path="/var/lib/kubelet/pods/3c210ff5-234b-495f-835e-768b3f5a5b88/volumes" Nov 01 04:12:18.528187 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:18.528158 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" path="/var/lib/kubelet/pods/902b46b1-e745-435e-bbfa-94735bee93bf/volumes" Nov 01 04:12:33.001573 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:33.001537 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:12:38.098352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:38.098311 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6"] Nov 01 04:12:38.201627 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:38.201593 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474"] Nov 01 04:12:41.760724 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.760694 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh"] Nov 01 04:12:41.761563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761541 6735 cpu_manager.go:414] "containerMap: removing container" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" Nov 01 04:12:41.761563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761561 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761575 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1f914c94-2cd3-4576-854b-6b0764e120d6" containerName="payload-pre-processing" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761581 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f914c94-2cd3-4576-854b-6b0764e120d6" containerName="payload-pre-processing" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761597 6735 cpu_manager.go:414] "containerMap: removing container" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761603 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761617 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761622 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761629 6735 cpu_manager.go:414] "containerMap: removing container" podUID="20da01e7-4942-4ced-a858-687b8d77ae18" containerName="payload-processing" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761633 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20da01e7-4942-4ced-a858-687b8d77ae18" containerName="payload-processing" Nov 01 04:12:41.761718 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761721 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f914c94-2cd3-4576-854b-6b0764e120d6" containerName="payload-pre-processing" Nov 01 04:12:41.761989 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761731 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="902b46b1-e745-435e-bbfa-94735bee93bf" containerName="istio-proxy" Nov 01 04:12:41.761989 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761738 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a37e982-6885-4ad9-a0f9-7604756f6612" containerName="istio-proxy" Nov 01 04:12:41.761989 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761745 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="20da01e7-4942-4ced-a858-687b8d77ae18" containerName="payload-processing" Nov 01 04:12:41.761989 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.761757 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c210ff5-234b-495f-835e-768b3f5a5b88" containerName="istio-proxy" Nov 01 04:12:41.765394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.765376 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:41.814354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.814320 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5b71cd-serving-cert\"" Nov 01 04:12:41.863563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.863529 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh"] Nov 01 04:12:41.910851 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.910818 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912227d8-0b7e-47c0-9585-46e708095369-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:41.911059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.910952 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-metrics-certs\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:41.911059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.910978 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4sjv\" (UniqueName: \"kubernetes.io/projected/912227d8-0b7e-47c0-9585-46e708095369-kube-api-access-s4sjv\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:41.911059 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:41.910996 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.013293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.012303 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912227d8-0b7e-47c0-9585-46e708095369-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.013293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.012473 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-metrics-certs\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.013293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.012973 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4sjv\" (UniqueName: \"kubernetes.io/projected/912227d8-0b7e-47c0-9585-46e708095369-kube-api-access-s4sjv\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.013293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.013189 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912227d8-0b7e-47c0-9585-46e708095369-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.015843 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.013385 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.016059 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:42.016011 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-serving-cert: secret "maas-api-e2e-ipp-a-5b71cd-serving-cert" not found Nov 01 04:12:42.016241 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:42.016229 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls podName:912227d8-0b7e-47c0-9585-46e708095369 nodeName:}" failed. No retries permitted until 2026-11-01 04:12:42.516205551 +0000 UTC m=+1774.105300171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls") pod "maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" (UID: "912227d8-0b7e-47c0-9585-46e708095369") : secret "maas-api-e2e-ipp-a-5b71cd-serving-cert" not found Nov 01 04:12:42.043358 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.043010 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-metrics-certs\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.071476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.071448 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4sjv\" (UniqueName: \"kubernetes.io/projected/912227d8-0b7e-47c0-9585-46e708095369-kube-api-access-s4sjv\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.550350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.550321 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.552981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.552956 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:42.679892 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:42.679853 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:43.051245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.051211 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh"] Nov 01 04:12:43.053162 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:12:43.053135 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912227d8_0b7e_47c0_9585_46e708095369.slice/crio-2aa8c17a4683b0042f1536da5bb63d2535f835e3183c9bd922239f370393f7fc WatchSource:0}: Error finding container 2aa8c17a4683b0042f1536da5bb63d2535f835e3183c9bd922239f370393f7fc: Status 404 returned error can't find the container with id 2aa8c17a4683b0042f1536da5bb63d2535f835e3183c9bd922239f370393f7fc Nov 01 04:12:43.100264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.100223 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" podUID="aeccb9e1-5fd7-457b-a11f-84f8baa26120" containerName="payload-processing" containerID="cri-o://d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada" gracePeriod=25 Nov 01 04:12:43.202447 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.202383 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" podUID="643eea33-f1c4-438b-a453-5ed690d5ca96" containerName="payload-pre-processing" containerID="cri-o://1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a" gracePeriod=25 Nov 01 04:12:43.377509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.377478 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:12:43.463072 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.463014 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeccb9e1-5fd7-457b-a11f-84f8baa26120-kube-api-access-sx9gn\" (UniqueName: \"kubernetes.io/projected/aeccb9e1-5fd7-457b-a11f-84f8baa26120-kube-api-access-sx9gn\") pod \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " Nov 01 04:12:43.463278 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.463218 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeccb9e1-5fd7-457b-a11f-84f8baa26120-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeccb9e1-5fd7-457b-a11f-84f8baa26120-plugins-config-volume\") pod \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\" (UID: \"aeccb9e1-5fd7-457b-a11f-84f8baa26120\") " Nov 01 04:12:43.463714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.463621 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeccb9e1-5fd7-457b-a11f-84f8baa26120-plugins-config-volume" pod "aeccb9e1-5fd7-457b-a11f-84f8baa26120" (UID: "aeccb9e1-5fd7-457b-a11f-84f8baa26120"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:12:43.465381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.465352 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeccb9e1-5fd7-457b-a11f-84f8baa26120-kube-api-access-sx9gn" pod "aeccb9e1-5fd7-457b-a11f-84f8baa26120" (UID: "aeccb9e1-5fd7-457b-a11f-84f8baa26120"). InnerVolumeSpecName "kube-api-access-sx9gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:43.477970 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.477944 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:12:43.564854 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.564820 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643eea33-f1c4-438b-a453-5ed690d5ca96-kube-api-access-b6ndd\" (UniqueName: \"kubernetes.io/projected/643eea33-f1c4-438b-a453-5ed690d5ca96-kube-api-access-b6ndd\") pod \"643eea33-f1c4-438b-a453-5ed690d5ca96\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " Nov 01 04:12:43.565091 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.565026 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643eea33-f1c4-438b-a453-5ed690d5ca96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643eea33-f1c4-438b-a453-5ed690d5ca96-plugins-config-volume\") pod \"643eea33-f1c4-438b-a453-5ed690d5ca96\" (UID: \"643eea33-f1c4-438b-a453-5ed690d5ca96\") " Nov 01 04:12:43.565369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.565353 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9gn\" (UniqueName: \"kubernetes.io/projected/aeccb9e1-5fd7-457b-a11f-84f8baa26120-kube-api-access-sx9gn\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:43.565441 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.565370 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeccb9e1-5fd7-457b-a11f-84f8baa26120-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:43.565441 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.565365 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643eea33-f1c4-438b-a453-5ed690d5ca96-plugins-config-volume" pod "643eea33-f1c4-438b-a453-5ed690d5ca96" (UID: "643eea33-f1c4-438b-a453-5ed690d5ca96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:12:43.567268 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.567219 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643eea33-f1c4-438b-a453-5ed690d5ca96-kube-api-access-b6ndd" pod "643eea33-f1c4-438b-a453-5ed690d5ca96" (UID: "643eea33-f1c4-438b-a453-5ed690d5ca96"). InnerVolumeSpecName "kube-api-access-b6ndd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:12:43.666566 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.666525 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6ndd\" (UniqueName: \"kubernetes.io/projected/643eea33-f1c4-438b-a453-5ed690d5ca96-kube-api-access-b6ndd\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:43.666566 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.666566 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643eea33-f1c4-438b-a453-5ed690d5ca96-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:12:43.714016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.713977 6735 generic.go:356] "Generic (PLEG): container finished" podID="643eea33-f1c4-438b-a453-5ed690d5ca96" containerID="1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a" exitCode=0 Nov 01 04:12:43.714237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.714030 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" Nov 01 04:12:43.714237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.714061 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" event={"ID":"643eea33-f1c4-438b-a453-5ed690d5ca96","Type":"ContainerDied","Data":"1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a"} Nov 01 04:12:43.714237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.714116 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474" event={"ID":"643eea33-f1c4-438b-a453-5ed690d5ca96","Type":"ContainerDied","Data":"41f4985f7f64e739e0d110809b2a4e4ec4cb6090b5caceff65ca02cd577f26dc"} Nov 01 04:12:43.714237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.714139 6735 scope.go:122] "RemoveContainer" containerID="1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a" Nov 01 04:12:43.715646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.715467 6735 generic.go:356] "Generic (PLEG): container finished" podID="aeccb9e1-5fd7-457b-a11f-84f8baa26120" containerID="d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada" exitCode=0 Nov 01 04:12:43.715646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.715528 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" Nov 01 04:12:43.715646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.715622 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" event={"ID":"aeccb9e1-5fd7-457b-a11f-84f8baa26120","Type":"ContainerDied","Data":"d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada"} Nov 01 04:12:43.715828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.715657 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6" event={"ID":"aeccb9e1-5fd7-457b-a11f-84f8baa26120","Type":"ContainerDied","Data":"a0b41c89eb5e02352cfee5a38d401ac900a2a3a4ee916545e05e655955e755a5"} Nov 01 04:12:43.717148 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.717122 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" event={"ID":"912227d8-0b7e-47c0-9585-46e708095369","Type":"ContainerStarted","Data":"2aa8c17a4683b0042f1536da5bb63d2535f835e3183c9bd922239f370393f7fc"} Nov 01 04:12:43.725104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.725078 6735 scope.go:122] "RemoveContainer" containerID="1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a" Nov 01 04:12:43.725416 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:43.725387 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a\": container with ID starting with 1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a not found: ID does not exist" containerID="1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a" Nov 01 04:12:43.725511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.725417 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a"} err="failed to get container status \"1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a\": rpc error: code = NotFound desc = could not find container \"1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a\": container with ID starting with 1e18565af4ed8018deeffac4eb4b9f6ea6483dafc88a42db3de2a605ae05e03a not found: ID does not exist" Nov 01 04:12:43.725511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.725441 6735 scope.go:122] "RemoveContainer" containerID="d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada" Nov 01 04:12:43.734961 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.734944 6735 scope.go:122] "RemoveContainer" containerID="d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada" Nov 01 04:12:43.735252 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:12:43.735232 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada\": container with ID starting with d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada not found: ID does not exist" containerID="d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada" Nov 01 04:12:43.735308 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.735260 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada"} err="failed to get container status \"d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada\": rpc error: code = NotFound desc = could not find container \"d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada\": container with ID starting with d79f576fefb9e98da23cc12b2ea9a308d209d83557ff1f5601d1bacd92caeada not found: ID does not exist" Nov 01 04:12:43.873028 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.872941 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6"] Nov 01 04:12:43.879267 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:43.879241 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-76f89d5c85-5q8z6"] Nov 01 04:12:44.009280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:44.009252 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474"] Nov 01 04:12:44.030163 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:44.030137 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-86b4b4bfff-ch474"] Nov 01 04:12:44.527572 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:44.527530 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643eea33-f1c4-438b-a453-5ed690d5ca96" path="/var/lib/kubelet/pods/643eea33-f1c4-438b-a453-5ed690d5ca96/volumes" Nov 01 04:12:44.528155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:44.528136 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeccb9e1-5fd7-457b-a11f-84f8baa26120" path="/var/lib/kubelet/pods/aeccb9e1-5fd7-457b-a11f-84f8baa26120/volumes" Nov 01 04:12:44.726108 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:44.726078 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" event={"ID":"912227d8-0b7e-47c0-9585-46e708095369","Type":"ContainerStarted","Data":"92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738"} Nov 01 04:12:44.726303 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:44.726208 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:12:44.840551 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:44.840447 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" podStartSLOduration=2.8949327719999998 podStartE2EDuration="3.84043139s" totalImagesPullingTime="945.498618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:12:44.83887862 +0000 UTC m=+1776.427973253" watchObservedRunningTime="2026-11-01 04:12:44.84043139 +0000 UTC m=+1776.429526017" Nov 01 04:12:50.737082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:12:50.737048 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:13:00.329629 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.329594 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh"] Nov 01 04:13:00.330204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.330189 6735 cpu_manager.go:414] "containerMap: removing container" podUID="643eea33-f1c4-438b-a453-5ed690d5ca96" containerName="payload-pre-processing" Nov 01 04:13:00.330266 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.330204 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643eea33-f1c4-438b-a453-5ed690d5ca96" containerName="payload-pre-processing" Nov 01 04:13:00.330266 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.330227 6735 cpu_manager.go:414] "containerMap: removing container" podUID="aeccb9e1-5fd7-457b-a11f-84f8baa26120" containerName="payload-processing" Nov 01 04:13:00.330266 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.330232 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeccb9e1-5fd7-457b-a11f-84f8baa26120" containerName="payload-processing" Nov 01 04:13:00.330389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.330313 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeccb9e1-5fd7-457b-a11f-84f8baa26120" containerName="payload-processing" Nov 01 04:13:00.330389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.330327 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="643eea33-f1c4-438b-a453-5ed690d5ca96" containerName="payload-pre-processing" Nov 01 04:13:00.333787 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.333767 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.376324 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.376291 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb"] Nov 01 04:13:00.379430 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379401 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2qz\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-kube-api-access-2k2qz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379556 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379459 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379598 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379544 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379616 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379688 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379675 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379724 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379709 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1003142-0404-4993-a9b0-8ddd5cb38f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379771 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379798 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.379961 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.379944 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.380185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.380169 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx"] Nov 01 04:13:00.380304 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.380289 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.383966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.383947 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.390488 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.390469 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qr8p5\"" Nov 01 04:13:00.430846 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.430807 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh"] Nov 01 04:13:00.480725 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.480695 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.480993 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.480736 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.480993 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.480771 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2qz\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-kube-api-access-2k2qz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.481283 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481222 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.481395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481295 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.481395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481328 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.481395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481358 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481442 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481487 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481529 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481556 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481586 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481637 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.481780 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482245 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482731 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482766 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482790 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482821 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1003142-0404-4993-a9b0-8ddd5cb38f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482873 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482897 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.482990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.482932 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffht\" (UniqueName: \"kubernetes.io/projected/bd9b484a-a7c6-438e-b741-13e153e3f2ff-kube-api-access-jffht\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.483786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483246 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.483786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483303 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.483786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483497 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.483786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483553 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.483786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483598 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2sh\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-kube-api-access-pg2sh\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.483786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483779 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.484153 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483815 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.484153 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.483889 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.484261 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.484212 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1003142-0404-4993-a9b0-8ddd5cb38f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.484317 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.484263 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.487073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.485228 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb"] Nov 01 04:13:00.487073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.485548 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.487073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.485844 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.500433 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.500405 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx"] Nov 01 04:13:00.538868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.538841 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2qz\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-kube-api-access-2k2qz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.540614 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.540583 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-j94nh\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.584861 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584789 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.584861 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584828 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.584861 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584847 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2sh\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-kube-api-access-pg2sh\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584864 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584881 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584945 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584966 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.584995 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585019 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585170 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585223 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585262 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585274 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585292 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585336 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585370 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585436 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585466 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585490 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585567 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jffht\" (UniqueName: \"kubernetes.io/projected/bd9b484a-a7c6-438e-b741-13e153e3f2ff-kube-api-access-jffht\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585578 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585637 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.585934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585801 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585908 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.585934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.585923 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.586524 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.586304 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.586524 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.586368 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.586739 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.586714 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.587884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.587856 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.587986 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.587853 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.588231 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.588210 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.588298 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.588284 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.642569 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.642537 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2sh\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-kube-api-access-pg2sh\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.642707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.642590 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9b484a-a7c6-438e-b741-13e153e3f2ff-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.642771 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.642722 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffht\" (UniqueName: \"kubernetes.io/projected/bd9b484a-a7c6-438e-b741-13e153e3f2ff-kube-api-access-jffht\") pod \"maas-default-gateway-openshift-default-5df48fcc99-p79qx\" (UID: \"bd9b484a-a7c6-438e-b741-13e153e3f2ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:00.642920 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.642906 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zhpnb\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.647919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.647898 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:00.697083 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.697027 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:00.702708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:00.702249 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:01.067902 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.067869 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh"] Nov 01 04:13:01.102642 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:01.102609 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1003142_0404_4993_a9b0_8ddd5cb38f40.slice/crio-f652dfd68aec88c118c5bca9e57e9c667adb966d6d9e2afd489ff3328ccf4c4a WatchSource:0}: Error finding container f652dfd68aec88c118c5bca9e57e9c667adb966d6d9e2afd489ff3328ccf4c4a: Status 404 returned error can't find the container with id f652dfd68aec88c118c5bca9e57e9c667adb966d6d9e2afd489ff3328ccf4c4a Nov 01 04:13:01.105010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.104978 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.105096 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.105067 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.105146 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.105105 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.116651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.116631 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb"] Nov 01 04:13:01.118250 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:01.118222 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf893cd_f7ae_43fc_b66f_0840ad6cc48e.slice/crio-a95567e41cb7f9e46d92059f929ee8d8d8b285bb5878c08afdb20c3bcd86030f WatchSource:0}: Error finding container a95567e41cb7f9e46d92059f929ee8d8d8b285bb5878c08afdb20c3bcd86030f: Status 404 returned error can't find the container with id a95567e41cb7f9e46d92059f929ee8d8d8b285bb5878c08afdb20c3bcd86030f Nov 01 04:13:01.122605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.122215 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.122605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.122296 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.122605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.122353 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.165073 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.165049 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx"] Nov 01 04:13:01.166748 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:01.166716 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9b484a_a7c6_438e_b741_13e153e3f2ff.slice/crio-9980f9fcacb67dc4124d3aa6890c384050cb1c9b3c6b1384a398678082ec8b9e WatchSource:0}: Error finding container 9980f9fcacb67dc4124d3aa6890c384050cb1c9b3c6b1384a398678082ec8b9e: Status 404 returned error can't find the container with id 9980f9fcacb67dc4124d3aa6890c384050cb1c9b3c6b1384a398678082ec8b9e Nov 01 04:13:01.169404 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.169366 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.169489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.169445 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.169489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.169478 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:01.842704 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.842666 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" event={"ID":"bd9b484a-a7c6-438e-b741-13e153e3f2ff","Type":"ContainerStarted","Data":"20d42fa94851d15373f6fa9ba92998388702001af2f5ca13c8218e3a8494bc6f"} Nov 01 04:13:01.843162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.842748 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" event={"ID":"bd9b484a-a7c6-438e-b741-13e153e3f2ff","Type":"ContainerStarted","Data":"9980f9fcacb67dc4124d3aa6890c384050cb1c9b3c6b1384a398678082ec8b9e"} Nov 01 04:13:01.844670 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.844642 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" event={"ID":"e1003142-0404-4993-a9b0-8ddd5cb38f40","Type":"ContainerStarted","Data":"77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d"} Nov 01 04:13:01.844804 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.844694 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" event={"ID":"e1003142-0404-4993-a9b0-8ddd5cb38f40","Type":"ContainerStarted","Data":"f652dfd68aec88c118c5bca9e57e9c667adb966d6d9e2afd489ff3328ccf4c4a"} Nov 01 04:13:01.883061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.880385 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" event={"ID":"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e","Type":"ContainerStarted","Data":"40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851"} Nov 01 04:13:01.883061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.880456 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" event={"ID":"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e","Type":"ContainerStarted","Data":"a95567e41cb7f9e46d92059f929ee8d8d8b285bb5878c08afdb20c3bcd86030f"} Nov 01 04:13:01.995135 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:01.995069 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podStartSLOduration=1.995028542 podStartE2EDuration="1.995028542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:01.994474714 +0000 UTC m=+1793.583569344" watchObservedRunningTime="2026-11-01 04:13:01.995028542 +0000 UTC m=+1793.584123172" Nov 01 04:13:02.122152 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.114479 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podStartSLOduration=2.114457049 podStartE2EDuration="2.114457049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:02.113534495 +0000 UTC m=+1793.702629136" watchObservedRunningTime="2026-11-01 04:13:02.114457049 +0000 UTC m=+1793.703551679" Nov 01 04:13:02.237091 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.236753 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podStartSLOduration=2.236733941 podStartE2EDuration="2.236733941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:02.236372043 +0000 UTC m=+1793.825466673" watchObservedRunningTime="2026-11-01 04:13:02.236733941 +0000 UTC m=+1793.825828571" Nov 01 04:13:02.651343 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.651304 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:02.651733 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.651705 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 04:13:02.655062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.651908 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 04:13:02.719066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.715305 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:02.719066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.715383 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:02.719066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.715516 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zhpnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 04:13:02.719066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.715566 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 04:13:02.719066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.715685 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p79qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 04:13:02.719066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:02.715722 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podUID="bd9b484a-a7c6-438e-b741-13e153e3f2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 04:13:03.043369 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:13:03.043332 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:13:03.649493 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:03.649452 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 04:13:03.649794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:03.649768 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 04:13:03.705893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:03.705854 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zhpnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 04:13:03.706259 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:03.706229 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 04:13:03.706874 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:03.706840 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p79qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 04:13:03.707050 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:03.707013 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podUID="bd9b484a-a7c6-438e-b741-13e153e3f2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 04:13:04.653066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:04.649485 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 04:13:04.653066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:04.649564 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 04:13:04.706063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:04.705235 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zhpnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 04:13:04.706063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:04.705294 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 04:13:04.706063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:04.705367 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p79qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 04:13:04.706063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:04.705389 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podUID="bd9b484a-a7c6-438e-b741-13e153e3f2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 04:13:05.653063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:05.649450 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 04:13:05.653063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:05.649518 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 04:13:05.708063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:05.706144 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p79qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 04:13:05.708063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:05.706208 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podUID="bd9b484a-a7c6-438e-b741-13e153e3f2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 04:13:05.708063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:05.706296 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zhpnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 04:13:05.708063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:05.706316 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 04:13:06.649473 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:06.649434 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 04:13:06.649812 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:06.649767 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 04:13:06.701620 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:06.701581 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zhpnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 04:13:06.702293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:06.702256 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 04:13:06.702794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:06.702775 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p79qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 04:13:06.702965 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:06.702945 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podUID="bd9b484a-a7c6-438e-b741-13e153e3f2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 04:13:07.648520 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:07.648482 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 04:13:07.648735 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:07.648558 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 04:13:07.698490 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:07.698447 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zhpnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 04:13:07.698678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:07.698518 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 04:13:07.703193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:07.703159 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p79qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 04:13:07.703532 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:07.703221 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podUID="bd9b484a-a7c6-438e-b741-13e153e3f2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 04:13:08.648928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:08.648894 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 04:13:08.649150 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:08.648956 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 04:13:08.698421 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:08.698381 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zhpnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 01 04:13:08.698578 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:08.698464 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 01 04:13:08.702553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:08.702518 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-p79qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 01 04:13:08.702687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:08.702562 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" podUID="bd9b484a-a7c6-438e-b741-13e153e3f2ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 01 04:13:09.738406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:09.738317 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:13:09.742507 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:09.742482 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:13:10.254087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.253723 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:10.254296 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.254279 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:10.257074 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.255655 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-p79qx" Nov 01 04:13:10.264925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.264900 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:10.266178 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.266154 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:10.267075 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.267054 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:13:10.275484 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.275431 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:10.275852 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.275834 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:10.276854 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:10.276835 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:13:17.858625 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:17.858538 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:13:21.499704 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:21.499659 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl"] Nov 01 04:13:21.500592 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:21.500555 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" podUID="62a4a366-e024-4bb5-87f0-848a646c136a" containerName="maas-api" containerID="cri-o://9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0" gracePeriod=30 Nov 01 04:13:21.995028 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:13:21.994987 6735 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a4a366_e024_4bb5_87f0_848a646c136a.slice/crio-9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0.scope\": RecentStats: unable to find data in memory cache]" Nov 01 04:13:22.788024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.787973 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:13:22.971423 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.971386 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a4a366-e024-4bb5-87f0-848a646c136a-kube-api-access-zlqjw\" (UniqueName: \"kubernetes.io/projected/62a4a366-e024-4bb5-87f0-848a646c136a-kube-api-access-zlqjw\") pod \"62a4a366-e024-4bb5-87f0-848a646c136a\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " Nov 01 04:13:22.971642 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.971453 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-metrics-certs\") pod \"62a4a366-e024-4bb5-87f0-848a646c136a\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " Nov 01 04:13:22.971642 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.971514 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls\") pod \"62a4a366-e024-4bb5-87f0-848a646c136a\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " Nov 01 04:13:22.971752 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.971675 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a4a366-e024-4bb5-87f0-848a646c136a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4a366-e024-4bb5-87f0-848a646c136a-openshift-service-ca\") pod \"62a4a366-e024-4bb5-87f0-848a646c136a\" (UID: \"62a4a366-e024-4bb5-87f0-848a646c136a\") " Nov 01 04:13:22.972263 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.972227 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a4a366-e024-4bb5-87f0-848a646c136a-openshift-service-ca" pod "62a4a366-e024-4bb5-87f0-848a646c136a" (UID: "62a4a366-e024-4bb5-87f0-848a646c136a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:13:22.974208 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.974186 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-metrics-certs" pod "62a4a366-e024-4bb5-87f0-848a646c136a" (UID: "62a4a366-e024-4bb5-87f0-848a646c136a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:13:22.974356 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.974329 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a4a366-e024-4bb5-87f0-848a646c136a-kube-api-access-zlqjw" pod "62a4a366-e024-4bb5-87f0-848a646c136a" (UID: "62a4a366-e024-4bb5-87f0-848a646c136a"). InnerVolumeSpecName "kube-api-access-zlqjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:13:22.976404 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:22.976382 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls" pod "62a4a366-e024-4bb5-87f0-848a646c136a" (UID: "62a4a366-e024-4bb5-87f0-848a646c136a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:13:23.073966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.073867 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4a366-e024-4bb5-87f0-848a646c136a-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:23.073966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.073902 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqjw\" (UniqueName: \"kubernetes.io/projected/62a4a366-e024-4bb5-87f0-848a646c136a-kube-api-access-zlqjw\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:23.073966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.073918 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:23.073966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.073929 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a4a366-e024-4bb5-87f0-848a646c136a-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:23.174826 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.174791 6735 generic.go:356] "Generic (PLEG): container finished" podID="62a4a366-e024-4bb5-87f0-848a646c136a" containerID="9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0" exitCode=0 Nov 01 04:13:23.175003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.174837 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" event={"ID":"62a4a366-e024-4bb5-87f0-848a646c136a","Type":"ContainerDied","Data":"9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0"} Nov 01 04:13:23.175003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.174906 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" event={"ID":"62a4a366-e024-4bb5-87f0-848a646c136a","Type":"ContainerDied","Data":"118502a47c2de646e582e4728edb8159c6085444dc4f5eef1c2d187b2e214938"} Nov 01 04:13:23.175003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.174932 6735 scope.go:122] "RemoveContainer" containerID="9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0" Nov 01 04:13:23.175003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.174853 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" Nov 01 04:13:23.185721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.185700 6735 scope.go:122] "RemoveContainer" containerID="9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0" Nov 01 04:13:23.186077 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:13:23.186054 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0\": container with ID starting with 9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0 not found: ID does not exist" containerID="9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0" Nov 01 04:13:23.186160 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.186085 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0"} err="failed to get container status \"9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0\": rpc error: code = NotFound desc = could not find container \"9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0\": container with ID starting with 9e8ea4deb7a6eccfcea721ead0421c0a527f932169169cf7c497d6e1bab2e4c0 not found: ID does not exist" Nov 01 04:13:23.330924 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.330834 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl"] Nov 01 04:13:23.376464 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:23.376427 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl"] Nov 01 04:13:24.532216 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:24.532185 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a4a366-e024-4bb5-87f0-848a646c136a" path="/var/lib/kubelet/pods/62a4a366-e024-4bb5-87f0-848a646c136a/volumes" Nov 01 04:13:25.547357 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:25.547317 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc980b-8467dfc596-gcxjl" podUID="62a4a366-e024-4bb5-87f0-848a646c136a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.100:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 04:13:29.213549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.213518 6735 generic.go:356] "Generic (PLEG): container finished" podID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerID="bbdb520cdda18265b0f29918040950a03f584822d0b8316d293b21f865b82e40" exitCode=0 Nov 01 04:13:29.214050 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.213589 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" event={"ID":"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc","Type":"ContainerDied","Data":"bbdb520cdda18265b0f29918040950a03f584822d0b8316d293b21f865b82e40"} Nov 01 04:13:29.214050 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.214026 6735 scope.go:122] "RemoveContainer" containerID="bbdb520cdda18265b0f29918040950a03f584822d0b8316d293b21f865b82e40" Nov 01 04:13:29.215501 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.215470 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:29.215613 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.215553 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:29.215613 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.215602 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:13:29.367239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.367206 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx"] Nov 01 04:13:29.368189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.368165 6735 cpu_manager.go:414] "containerMap: removing container" podUID="62a4a366-e024-4bb5-87f0-848a646c136a" containerName="maas-api" Nov 01 04:13:29.368189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.368185 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a4a366-e024-4bb5-87f0-848a646c136a" containerName="maas-api" Nov 01 04:13:29.368455 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.368308 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a4a366-e024-4bb5-87f0-848a646c136a" containerName="maas-api" Nov 01 04:13:29.370809 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.370790 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:29.419896 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.419868 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-157fed\"" Nov 01 04:13:29.430955 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.430895 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx"] Nov 01 04:13:29.461016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.460982 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-157fed-dockercfg-d9tl4\"" Nov 01 04:13:29.473790 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.473707 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b102f43a-f1e6-4295-a344-f336c6e54c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:29.473790 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.473758 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djrj5\" (UniqueName: \"kubernetes.io/projected/b102f43a-f1e6-4295-a344-f336c6e54c5b-kube-api-access-djrj5\") pod \"payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:29.575115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.575079 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b102f43a-f1e6-4295-a344-f336c6e54c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:29.575330 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.575143 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djrj5\" (UniqueName: \"kubernetes.io/projected/b102f43a-f1e6-4295-a344-f336c6e54c5b-kube-api-access-djrj5\") pod \"payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:29.575964 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.575943 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b102f43a-f1e6-4295-a344-f336c6e54c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:29.633557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.633526 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djrj5\" (UniqueName: \"kubernetes.io/projected/b102f43a-f1e6-4295-a344-f336c6e54c5b-kube-api-access-djrj5\") pod \"payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:29.685750 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:29.685720 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:30.051585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.051557 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx"] Nov 01 04:13:30.052623 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:30.052596 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb102f43a_f1e6_4295_a344_f336c6e54c5b.slice/crio-e967d9f6e577eee75e8e0941f798028ce16d0208ff30f4b7f79bd273af860f20 WatchSource:0}: Error finding container e967d9f6e577eee75e8e0941f798028ce16d0208ff30f4b7f79bd273af860f20: Status 404 returned error can't find the container with id e967d9f6e577eee75e8e0941f798028ce16d0208ff30f4b7f79bd273af860f20 Nov 01 04:13:30.151359 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.151318 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9"] Nov 01 04:13:30.154921 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.154898 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.183758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.183436 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956fe986-cddf-4077-b9df-4a4a5b93a1d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.183758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.183643 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfk2g\" (UniqueName: \"kubernetes.io/projected/956fe986-cddf-4077-b9df-4a4a5b93a1d3-kube-api-access-rfk2g\") pod \"payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.211119 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.211086 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9"] Nov 01 04:13:30.223635 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.223596 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" event={"ID":"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc","Type":"ContainerStarted","Data":"6a951528afff03326deed920d0c0b609b98636aaf5de7959feacfd913cef138e"} Nov 01 04:13:30.225014 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.224988 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" event={"ID":"b102f43a-f1e6-4295-a344-f336c6e54c5b","Type":"ContainerStarted","Data":"e967d9f6e577eee75e8e0941f798028ce16d0208ff30f4b7f79bd273af860f20"} Nov 01 04:13:30.285089 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.285048 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfk2g\" (UniqueName: \"kubernetes.io/projected/956fe986-cddf-4077-b9df-4a4a5b93a1d3-kube-api-access-rfk2g\") pod \"payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.285318 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.285126 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956fe986-cddf-4077-b9df-4a4a5b93a1d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.285885 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.285862 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956fe986-cddf-4077-b9df-4a4a5b93a1d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.288882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.288861 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:13:30.289748 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.289727 6735 patch_prober.go:29] interesting pod/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 04:13:30.289830 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.289769 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 04:13:30.342581 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.342515 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfk2g\" (UniqueName: \"kubernetes.io/projected/956fe986-cddf-4077-b9df-4a4a5b93a1d3-kube-api-access-rfk2g\") pod \"payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.470585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.470303 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:30.953060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:30.951949 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9"] Nov 01 04:13:30.978392 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:30.978360 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956fe986_cddf_4077_b9df_4a4a5b93a1d3.slice/crio-4e10293fac6367fd3a81f3fb46bacdbff7df1870281174c986baaebbc38a5e46 WatchSource:0}: Error finding container 4e10293fac6367fd3a81f3fb46bacdbff7df1870281174c986baaebbc38a5e46: Status 404 returned error can't find the container with id 4e10293fac6367fd3a81f3fb46bacdbff7df1870281174c986baaebbc38a5e46 Nov 01 04:13:31.232623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:31.232584 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" event={"ID":"956fe986-cddf-4077-b9df-4a4a5b93a1d3","Type":"ContainerStarted","Data":"4e10293fac6367fd3a81f3fb46bacdbff7df1870281174c986baaebbc38a5e46"} Nov 01 04:13:31.289313 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:31.289282 6735 patch_prober.go:29] interesting pod/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 04:13:31.289473 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:31.289352 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 04:13:32.289055 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:32.288993 6735 patch_prober.go:29] interesting pod/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 04:13:32.289555 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:32.289094 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 04:13:33.008020 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:13:33.007987 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:13:33.288979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:33.288888 6735 patch_prober.go:29] interesting pod/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 04:13:33.288979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:33.288937 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 04:13:34.318071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.293819 6735 patch_prober.go:29] interesting pod/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 04:13:34.318071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.293873 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 04:13:34.428794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.428636 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" event={"ID":"b102f43a-f1e6-4295-a344-f336c6e54c5b","Type":"ContainerStarted","Data":"6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8"} Nov 01 04:13:34.430174 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.429642 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:34.435812 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.435068 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" event={"ID":"956fe986-cddf-4077-b9df-4a4a5b93a1d3","Type":"ContainerStarted","Data":"4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612"} Nov 01 04:13:34.435812 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.435755 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:34.673116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.667606 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" podStartSLOduration=2.482438887 podStartE2EDuration="5.667586746s" totalImagesPullingTime="3.185147859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:34.555265525 +0000 UTC m=+1826.144360158" watchObservedRunningTime="2026-11-01 04:13:34.667586746 +0000 UTC m=+1826.256681373" Nov 01 04:13:34.684851 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:34.684779 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" podStartSLOduration=2.193800289 podStartE2EDuration="4.684760548s" totalImagesPullingTime="2.490960259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:34.66449002 +0000 UTC m=+1826.253584652" watchObservedRunningTime="2026-11-01 04:13:34.684760548 +0000 UTC m=+1826.273855180" Nov 01 04:13:35.289288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:35.289251 6735 patch_prober.go:29] interesting pod/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 04:13:35.289497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:35.289311 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 04:13:36.255720 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:36.255689 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:13:36.294627 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:36.294596 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:13:36.448905 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:36.448876 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:13:41.939895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:41.939855 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t"] Nov 01 04:13:41.945331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:41.945311 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.048090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.042388 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-131718-dockercfg-27rk7\"" Nov 01 04:13:42.048090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.042533 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-131718\"" Nov 01 04:13:42.051709 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.051677 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t"] Nov 01 04:13:42.068899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.068580 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2274\" (UniqueName: \"kubernetes.io/projected/4222c967-9ba8-446e-bb84-787570b73e28-kube-api-access-c2274\") pod \"payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.068899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.068658 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4222c967-9ba8-446e-bb84-787570b73e28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.170375 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.170346 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2274\" (UniqueName: \"kubernetes.io/projected/4222c967-9ba8-446e-bb84-787570b73e28-kube-api-access-c2274\") pod \"payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.170556 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.170406 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4222c967-9ba8-446e-bb84-787570b73e28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.171159 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.171137 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4222c967-9ba8-446e-bb84-787570b73e28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.228093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.228065 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2274\" (UniqueName: \"kubernetes.io/projected/4222c967-9ba8-446e-bb84-787570b73e28-kube-api-access-c2274\") pod \"payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.258947 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.258913 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:42.631513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:42.631440 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t"] Nov 01 04:13:42.633319 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:42.633293 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4222c967_9ba8_446e_bb84_787570b73e28.slice/crio-286a20780ac4f6073083f314046815aacce71c94dbc12c68472dc76fe5ac5afc WatchSource:0}: Error finding container 286a20780ac4f6073083f314046815aacce71c94dbc12c68472dc76fe5ac5afc: Status 404 returned error can't find the container with id 286a20780ac4f6073083f314046815aacce71c94dbc12c68472dc76fe5ac5afc Nov 01 04:13:43.520181 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:43.520143 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" event={"ID":"4222c967-9ba8-446e-bb84-787570b73e28","Type":"ContainerStarted","Data":"4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12"} Nov 01 04:13:43.520643 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:43.520191 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" event={"ID":"4222c967-9ba8-446e-bb84-787570b73e28","Type":"ContainerStarted","Data":"286a20780ac4f6073083f314046815aacce71c94dbc12c68472dc76fe5ac5afc"} Nov 01 04:13:43.520643 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:43.520265 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:43.630179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:43.630129 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" podStartSLOduration=2.630115524 podStartE2EDuration="2.630115524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:43.628087687 +0000 UTC m=+1835.217182317" watchObservedRunningTime="2026-11-01 04:13:43.630115524 +0000 UTC m=+1835.219210153" Nov 01 04:13:46.450265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:46.450231 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:13:46.450711 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:46.450384 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:13:46.999789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:46.999751 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk"] Nov 01 04:13:47.089478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:47.089436 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7"] Nov 01 04:13:52.001815 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.000111 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" podUID="eb9ad832-5e17-4b89-b8f5-62665639f4ed" containerName="payload-processing" containerID="cri-o://dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d" gracePeriod=25 Nov 01 04:13:52.090550 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.090506 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" podUID="13819afa-cd9e-4f16-b204-ad15d67a8d5d" containerName="payload-pre-processing" containerID="cri-o://55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53" gracePeriod=25 Nov 01 04:13:52.326070 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.324874 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:13:52.375192 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.375161 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:13:52.395024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.394999 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:13:52.455147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455114 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb9ad832-5e17-4b89-b8f5-62665639f4ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb9ad832-5e17-4b89-b8f5-62665639f4ed-plugins-config-volume\") pod \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " Nov 01 04:13:52.455331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455189 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13819afa-cd9e-4f16-b204-ad15d67a8d5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13819afa-cd9e-4f16-b204-ad15d67a8d5d-plugins-config-volume\") pod \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " Nov 01 04:13:52.455331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455250 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13819afa-cd9e-4f16-b204-ad15d67a8d5d-kube-api-access-xtdnj\" (UniqueName: \"kubernetes.io/projected/13819afa-cd9e-4f16-b204-ad15d67a8d5d-kube-api-access-xtdnj\") pod \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\" (UID: \"13819afa-cd9e-4f16-b204-ad15d67a8d5d\") " Nov 01 04:13:52.455447 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455389 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9ad832-5e17-4b89-b8f5-62665639f4ed-kube-api-access-pfh47\" (UniqueName: \"kubernetes.io/projected/eb9ad832-5e17-4b89-b8f5-62665639f4ed-kube-api-access-pfh47\") pod \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\" (UID: \"eb9ad832-5e17-4b89-b8f5-62665639f4ed\") " Nov 01 04:13:52.455574 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455547 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9ad832-5e17-4b89-b8f5-62665639f4ed-plugins-config-volume" pod "eb9ad832-5e17-4b89-b8f5-62665639f4ed" (UID: "eb9ad832-5e17-4b89-b8f5-62665639f4ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:13:52.455633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455557 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13819afa-cd9e-4f16-b204-ad15d67a8d5d-plugins-config-volume" pod "13819afa-cd9e-4f16-b204-ad15d67a8d5d" (UID: "13819afa-cd9e-4f16-b204-ad15d67a8d5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:13:52.455870 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455847 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb9ad832-5e17-4b89-b8f5-62665639f4ed-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:52.455870 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.455870 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13819afa-cd9e-4f16-b204-ad15d67a8d5d-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:52.457684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.457661 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9ad832-5e17-4b89-b8f5-62665639f4ed-kube-api-access-pfh47" pod "eb9ad832-5e17-4b89-b8f5-62665639f4ed" (UID: "eb9ad832-5e17-4b89-b8f5-62665639f4ed"). InnerVolumeSpecName "kube-api-access-pfh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:13:52.457684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.457676 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13819afa-cd9e-4f16-b204-ad15d67a8d5d-kube-api-access-xtdnj" pod "13819afa-cd9e-4f16-b204-ad15d67a8d5d" (UID: "13819afa-cd9e-4f16-b204-ad15d67a8d5d"). InnerVolumeSpecName "kube-api-access-xtdnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:13:52.557659 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.557635 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdnj\" (UniqueName: \"kubernetes.io/projected/13819afa-cd9e-4f16-b204-ad15d67a8d5d-kube-api-access-xtdnj\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:52.557659 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.557658 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfh47\" (UniqueName: \"kubernetes.io/projected/eb9ad832-5e17-4b89-b8f5-62665639f4ed-kube-api-access-pfh47\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:13:52.581695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.581667 6735 generic.go:356] "Generic (PLEG): container finished" podID="eb9ad832-5e17-4b89-b8f5-62665639f4ed" containerID="dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d" exitCode=0 Nov 01 04:13:52.581811 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.581780 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" Nov 01 04:13:52.581811 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.581784 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" event={"ID":"eb9ad832-5e17-4b89-b8f5-62665639f4ed","Type":"ContainerDied","Data":"dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d"} Nov 01 04:13:52.581916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.581837 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk" event={"ID":"eb9ad832-5e17-4b89-b8f5-62665639f4ed","Type":"ContainerDied","Data":"5827031eab872afc43bbb1673fb56814d9c6717f1f9368cf126cb302dfa9325e"} Nov 01 04:13:52.581916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.581866 6735 scope.go:122] "RemoveContainer" containerID="dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d" Nov 01 04:13:52.583248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.583225 6735 generic.go:356] "Generic (PLEG): container finished" podID="13819afa-cd9e-4f16-b204-ad15d67a8d5d" containerID="55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53" exitCode=0 Nov 01 04:13:52.583466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.583282 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" Nov 01 04:13:52.583466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.583324 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" event={"ID":"13819afa-cd9e-4f16-b204-ad15d67a8d5d","Type":"ContainerDied","Data":"55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53"} Nov 01 04:13:52.583466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.583402 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7" event={"ID":"13819afa-cd9e-4f16-b204-ad15d67a8d5d","Type":"ContainerDied","Data":"2173eb1219b993ca85cb53a1c9b5cccce9c5533fb35cbee8a370d691500337c8"} Nov 01 04:13:52.592111 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.592084 6735 scope.go:122] "RemoveContainer" containerID="dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d" Nov 01 04:13:52.592395 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:13:52.592374 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d\": container with ID starting with dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d not found: ID does not exist" containerID="dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d" Nov 01 04:13:52.592460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.592403 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d"} err="failed to get container status \"dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d\": rpc error: code = NotFound desc = could not find container \"dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d\": container with ID starting with dba446342e2f3d0297fa77afd4b2a8d273f056947fa1c919d54cc23c65256f4d not found: ID does not exist" Nov 01 04:13:52.592460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.592421 6735 scope.go:122] "RemoveContainer" containerID="55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53" Nov 01 04:13:52.602445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.602420 6735 scope.go:122] "RemoveContainer" containerID="55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53" Nov 01 04:13:52.602783 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:13:52.602759 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53\": container with ID starting with 55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53 not found: ID does not exist" containerID="55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53" Nov 01 04:13:52.602879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.602789 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53"} err="failed to get container status \"55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53\": rpc error: code = NotFound desc = could not find container \"55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53\": container with ID starting with 55e861966dd2df029004e3cd2f4261adb39ebf9fc6570e204e9d4d14c8a0ac53 not found: ID does not exist" Nov 01 04:13:52.717619 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.717590 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7"] Nov 01 04:13:52.736139 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.736109 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5b71cd-566999975-cxrq7"] Nov 01 04:13:52.860397 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.860313 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk"] Nov 01 04:13:52.873264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:52.873232 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5b71cd-7848f894d9-h8qqk"] Nov 01 04:13:54.527694 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:54.527659 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13819afa-cd9e-4f16-b204-ad15d67a8d5d" path="/var/lib/kubelet/pods/13819afa-cd9e-4f16-b204-ad15d67a8d5d/volumes" Nov 01 04:13:54.528198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:54.528168 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb9ad832-5e17-4b89-b8f5-62665639f4ed" path="/var/lib/kubelet/pods/eb9ad832-5e17-4b89-b8f5-62665639f4ed/volumes" Nov 01 04:13:54.530000 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:54.529980 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:13:54.632076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:54.632022 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx"] Nov 01 04:13:54.632272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:54.632250 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" containerID="cri-o://aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6" gracePeriod=30 Nov 01 04:13:56.015077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.015022 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p"] Nov 01 04:13:56.016017 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.015995 6735 cpu_manager.go:414] "containerMap: removing container" podUID="13819afa-cd9e-4f16-b204-ad15d67a8d5d" containerName="payload-pre-processing" Nov 01 04:13:56.016097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.016021 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13819afa-cd9e-4f16-b204-ad15d67a8d5d" containerName="payload-pre-processing" Nov 01 04:13:56.016097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.016090 6735 cpu_manager.go:414] "containerMap: removing container" podUID="eb9ad832-5e17-4b89-b8f5-62665639f4ed" containerName="payload-processing" Nov 01 04:13:56.016170 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.016100 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb9ad832-5e17-4b89-b8f5-62665639f4ed" containerName="payload-processing" Nov 01 04:13:56.016260 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.016246 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="13819afa-cd9e-4f16-b204-ad15d67a8d5d" containerName="payload-pre-processing" Nov 01 04:13:56.016295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.016272 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb9ad832-5e17-4b89-b8f5-62665639f4ed" containerName="payload-processing" Nov 01 04:13:56.020305 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.020283 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.069919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.069886 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-535e84\"" Nov 01 04:13:56.072710 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.072681 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p"] Nov 01 04:13:56.169065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.155164 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ls8\" (UniqueName: \"kubernetes.io/projected/de3edd7b-9086-42b8-a4ad-c9575db43548-kube-api-access-t7ls8\") pod \"payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.169065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.155391 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3edd7b-9086-42b8-a4ad-c9575db43548-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.235218 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.235147 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh"] Nov 01 04:13:56.241370 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.241341 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.260205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.260166 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ls8\" (UniqueName: \"kubernetes.io/projected/de3edd7b-9086-42b8-a4ad-c9575db43548-kube-api-access-t7ls8\") pod \"payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.260394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.260363 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3edd7b-9086-42b8-a4ad-c9575db43548-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.261517 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.261489 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3edd7b-9086-42b8-a4ad-c9575db43548-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.261716 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.261696 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh"] Nov 01 04:13:56.290992 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.290953 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-535e84-dockercfg-7589f\"" Nov 01 04:13:56.315640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.315615 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ls8\" (UniqueName: \"kubernetes.io/projected/de3edd7b-9086-42b8-a4ad-c9575db43548-kube-api-access-t7ls8\") pod \"payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.334218 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.334196 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:56.361584 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.361552 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.361846 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.361704 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvd9q\" (UniqueName: \"kubernetes.io/projected/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-kube-api-access-lvd9q\") pod \"payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.463504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.463467 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvd9q\" (UniqueName: \"kubernetes.io/projected/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-kube-api-access-lvd9q\") pod \"payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.463660 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.463578 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.464255 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.464235 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.521791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.521718 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvd9q\" (UniqueName: \"kubernetes.io/projected/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-kube-api-access-lvd9q\") pod \"payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.577006 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.576973 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:56.704865 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.704837 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p"] Nov 01 04:13:56.705631 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:56.705606 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3edd7b_9086_42b8_a4ad_c9575db43548.slice/crio-eda8a898a7b3289335a36903c771579de295cbc0465757f4c1da5061f3c45781 WatchSource:0}: Error finding container eda8a898a7b3289335a36903c771579de295cbc0465757f4c1da5061f3c45781: Status 404 returned error can't find the container with id eda8a898a7b3289335a36903c771579de295cbc0465757f4c1da5061f3c45781 Nov 01 04:13:56.961174 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:56.961149 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh"] Nov 01 04:13:56.962937 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:13:56.962909 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dd5772_0c1e_40e8_8709_d3b8ec86ddba.slice/crio-2c71a52d0218ab2b5b6cff0254aba8e0facd1a4d1282f31f57bbc0341c80f0b8 WatchSource:0}: Error finding container 2c71a52d0218ab2b5b6cff0254aba8e0facd1a4d1282f31f57bbc0341c80f0b8: Status 404 returned error can't find the container with id 2c71a52d0218ab2b5b6cff0254aba8e0facd1a4d1282f31f57bbc0341c80f0b8 Nov 01 04:13:57.625482 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.625444 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" event={"ID":"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba","Type":"ContainerStarted","Data":"e2d2c6b7fe878dcf9ecf0b73652814c1d2ee23da0ad8ab158602ac9efa5b6b20"} Nov 01 04:13:57.625908 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.625534 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" event={"ID":"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba","Type":"ContainerStarted","Data":"2c71a52d0218ab2b5b6cff0254aba8e0facd1a4d1282f31f57bbc0341c80f0b8"} Nov 01 04:13:57.625908 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.625564 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:13:57.627558 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.627528 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" event={"ID":"de3edd7b-9086-42b8-a4ad-c9575db43548","Type":"ContainerStarted","Data":"1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d"} Nov 01 04:13:57.627692 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.627571 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" event={"ID":"de3edd7b-9086-42b8-a4ad-c9575db43548","Type":"ContainerStarted","Data":"eda8a898a7b3289335a36903c771579de295cbc0465757f4c1da5061f3c45781"} Nov 01 04:13:57.627692 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.627674 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:13:57.734786 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.734734 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" podStartSLOduration=1.7347154 podStartE2EDuration="1.7347154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:57.734165232 +0000 UTC m=+1849.323259861" watchObservedRunningTime="2026-11-01 04:13:57.7347154 +0000 UTC m=+1849.323810025" Nov 01 04:13:57.845086 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:57.845007 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" podStartSLOduration=2.844987461 podStartE2EDuration="2.844987461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:13:57.842711094 +0000 UTC m=+1849.431805718" watchObservedRunningTime="2026-11-01 04:13:57.844987461 +0000 UTC m=+1849.434082090" Nov 01 04:13:58.997389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:58.997352 6735 patch_prober.go:29] interesting pod/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:13:58.997760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:58.997408 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:13:59.901719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:13:59.901661 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:14:00.015945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.015905 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-socket\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.015965 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-envoy\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016005 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3642c443-e753-4efe-871f-e55be1536776-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3642c443-e753-4efe-871f-e55be1536776-istiod-ca-cert\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016064 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-credential-socket\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016107 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-certs\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016182 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-data\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016248 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-socket" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016259 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-kube-api-access-zmdqv\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-kube-api-access-zmdqv\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016360 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-credential-socket" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016405 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3642c443-e753-4efe-871f-e55be1536776-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3642c443-e753-4efe-871f-e55be1536776-istio-podinfo\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016448 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-certs" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:14:00.016901 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016483 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3642c443-e753-4efe-871f-e55be1536776-istiod-ca-cert" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:14:00.016901 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016515 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-istio-token\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-istio-token\") pod \"3642c443-e753-4efe-871f-e55be1536776\" (UID: \"3642c443-e753-4efe-871f-e55be1536776\") " Nov 01 04:14:00.016901 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.016535 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-data" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:14:00.017367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.017337 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.017454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.017367 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3642c443-e753-4efe-871f-e55be1536776-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.017454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.017384 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.017454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.017399 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.017454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.017414 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.018415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.018390 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-envoy" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:14:00.018632 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.018618 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3642c443-e753-4efe-871f-e55be1536776-istio-podinfo" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:14:00.019064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.019023 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-kube-api-access-zmdqv" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "kube-api-access-zmdqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:14:00.019064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.019049 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-istio-token" pod "3642c443-e753-4efe-871f-e55be1536776" (UID: "3642c443-e753-4efe-871f-e55be1536776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:14:00.118653 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.118619 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3642c443-e753-4efe-871f-e55be1536776-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.118653 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.118652 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdqv\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-kube-api-access-zmdqv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.118847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.118667 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3642c443-e753-4efe-871f-e55be1536776-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.118847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.118678 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3642c443-e753-4efe-871f-e55be1536776-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:14:00.651458 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.651421 6735 generic.go:356] "Generic (PLEG): container finished" podID="3642c443-e753-4efe-871f-e55be1536776" containerID="aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6" exitCode=0 Nov 01 04:14:00.651672 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.651465 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" event={"ID":"3642c443-e753-4efe-871f-e55be1536776","Type":"ContainerDied","Data":"aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6"} Nov 01 04:14:00.651672 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.651490 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" Nov 01 04:14:00.651672 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.651526 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx" event={"ID":"3642c443-e753-4efe-871f-e55be1536776","Type":"ContainerDied","Data":"b700b7322bc1acd79210a48ff319fc1ae075655b184b4d1119bad402ceacfec6"} Nov 01 04:14:00.651672 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.651553 6735 scope.go:122] "RemoveContainer" containerID="aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6" Nov 01 04:14:00.666868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.665800 6735 scope.go:122] "RemoveContainer" containerID="aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6" Nov 01 04:14:00.666868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.665910 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw"] Nov 01 04:14:00.666868 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:14:00.666213 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6\": container with ID starting with aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6 not found: ID does not exist" containerID="aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6" Nov 01 04:14:00.666868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.666238 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6"} err="failed to get container status \"aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6\": rpc error: code = NotFound desc = could not find container \"aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6\": container with ID starting with aa9a5c70b740d7fbb49349053ecfa7e3982e0a7be5b30ad82fde65c58062d5e6 not found: ID does not exist" Nov 01 04:14:00.666868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.666804 6735 cpu_manager.go:414] "containerMap: removing container" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" Nov 01 04:14:00.666868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.666820 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" Nov 01 04:14:00.667200 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.666987 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="3642c443-e753-4efe-871f-e55be1536776" containerName="istio-proxy" Nov 01 04:14:00.672379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.672359 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.721704 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.721677 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d683e-dockercfg-vl8fw\"" Nov 01 04:14:00.721880 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.721723 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d683e\"" Nov 01 04:14:00.724539 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.724516 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw"] Nov 01 04:14:00.740268 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.740237 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lg4k\" (UniqueName: \"kubernetes.io/projected/5c0775ed-89fd-4eb8-86ae-3338a74051a2-kube-api-access-4lg4k\") pod \"payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.740404 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.740387 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0775ed-89fd-4eb8-86ae-3338a74051a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.827058 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.825010 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68"] Nov 01 04:14:00.843102 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.836219 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx"] Nov 01 04:14:00.843102 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.836262 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-fcpmx"] Nov 01 04:14:00.843102 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.836364 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:00.864102 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.859570 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0775ed-89fd-4eb8-86ae-3338a74051a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.883565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.883177 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68"] Nov 01 04:14:00.895837 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.895796 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0775ed-89fd-4eb8-86ae-3338a74051a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.896230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.896196 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lg4k\" (UniqueName: \"kubernetes.io/projected/5c0775ed-89fd-4eb8-86ae-3338a74051a2-kube-api-access-4lg4k\") pod \"payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.954339 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.954306 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lg4k\" (UniqueName: \"kubernetes.io/projected/5c0775ed-89fd-4eb8-86ae-3338a74051a2-kube-api-access-4lg4k\") pod \"payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.986640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.986609 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:00.998172 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.998148 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895cdd1f-c503-41e1-a4e6-383be3fb695a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:00.998295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:00.998278 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bss6j\" (UniqueName: \"kubernetes.io/projected/895cdd1f-c503-41e1-a4e6-383be3fb695a-kube-api-access-bss6j\") pod \"payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:01.099389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.099307 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895cdd1f-c503-41e1-a4e6-383be3fb695a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:01.099751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.099394 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bss6j\" (UniqueName: \"kubernetes.io/projected/895cdd1f-c503-41e1-a4e6-383be3fb695a-kube-api-access-bss6j\") pod \"payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:01.100083 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.100063 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895cdd1f-c503-41e1-a4e6-383be3fb695a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:01.156991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.156959 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bss6j\" (UniqueName: \"kubernetes.io/projected/895cdd1f-c503-41e1-a4e6-383be3fb695a-kube-api-access-bss6j\") pod \"payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:01.181002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.180977 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:01.361318 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.361286 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw"] Nov 01 04:14:01.362879 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:14:01.362851 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0775ed_89fd_4eb8_86ae_3338a74051a2.slice/crio-726909b376b53b9f6f02288cf9b59f6e1e8d459ed36727d1131ce6d5f29784f0 WatchSource:0}: Error finding container 726909b376b53b9f6f02288cf9b59f6e1e8d459ed36727d1131ce6d5f29784f0: Status 404 returned error can't find the container with id 726909b376b53b9f6f02288cf9b59f6e1e8d459ed36727d1131ce6d5f29784f0 Nov 01 04:14:01.551569 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.551538 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68"] Nov 01 04:14:01.553269 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:14:01.553244 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895cdd1f_c503_41e1_a4e6_383be3fb695a.slice/crio-dce15c1c381279788c344880e2c1496aaef0222234c04196547ba9939ddd1a84 WatchSource:0}: Error finding container dce15c1c381279788c344880e2c1496aaef0222234c04196547ba9939ddd1a84: Status 404 returned error can't find the container with id dce15c1c381279788c344880e2c1496aaef0222234c04196547ba9939ddd1a84 Nov 01 04:14:01.659374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.659341 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" event={"ID":"895cdd1f-c503-41e1-a4e6-383be3fb695a","Type":"ContainerStarted","Data":"d1ea50ee477f37d009abae6a6ae68624960eba2eb910fd5968b4b8e5205a6224"} Nov 01 04:14:01.659522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.659392 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" event={"ID":"895cdd1f-c503-41e1-a4e6-383be3fb695a","Type":"ContainerStarted","Data":"dce15c1c381279788c344880e2c1496aaef0222234c04196547ba9939ddd1a84"} Nov 01 04:14:01.659522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.659430 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:01.661144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.661108 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" event={"ID":"5c0775ed-89fd-4eb8-86ae-3338a74051a2","Type":"ContainerStarted","Data":"43fda6e2bda2b5401ebf3b5fc20798e9f853e91e03fc49d2fcfadd87a422bdbe"} Nov 01 04:14:01.661281 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.661169 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" event={"ID":"5c0775ed-89fd-4eb8-86ae-3338a74051a2","Type":"ContainerStarted","Data":"726909b376b53b9f6f02288cf9b59f6e1e8d459ed36727d1131ce6d5f29784f0"} Nov 01 04:14:01.661281 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.661202 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:01.769877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.769719 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" podStartSLOduration=1.7697012239999999 podStartE2EDuration="1.769701224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:14:01.767626597 +0000 UTC m=+1853.356721238" watchObservedRunningTime="2026-11-01 04:14:01.769701224 +0000 UTC m=+1853.358795855" Nov 01 04:14:01.895696 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:01.891559 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" podStartSLOduration=1.891536452 podStartE2EDuration="1.891536452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:14:01.886483634 +0000 UTC m=+1853.475578264" watchObservedRunningTime="2026-11-01 04:14:01.891536452 +0000 UTC m=+1853.480631084" Nov 01 04:14:02.527237 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:02.527207 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3642c443-e753-4efe-871f-e55be1536776" path="/var/lib/kubelet/pods/3642c443-e753-4efe-871f-e55be1536776/volumes" Nov 01 04:14:03.025299 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:14:03.025257 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:14:08.637572 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:08.637542 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:14:08.638090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:08.637620 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:14:12.671990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:12.671963 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:14:12.672460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:12.672051 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:14:33.001963 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:14:33.001930 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:14:51.686238 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.686202 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n"] Nov 01 04:14:51.700564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.700537 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.746734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.746702 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n"] Nov 01 04:14:51.790396 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.790359 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-535e84-serving-cert\"" Nov 01 04:14:51.825515 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.825471 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41574812-6157-45f7-a67b-e952c758491b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.825684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.825601 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-metrics-certs\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.825684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.825633 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhtl\" (UniqueName: \"kubernetes.io/projected/41574812-6157-45f7-a67b-e952c758491b-kube-api-access-srhtl\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.825807 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.825786 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.953122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.948535 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41574812-6157-45f7-a67b-e952c758491b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.953122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.948634 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-metrics-certs\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.953122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.948663 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhtl\" (UniqueName: \"kubernetes.io/projected/41574812-6157-45f7-a67b-e952c758491b-kube-api-access-srhtl\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.953122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.948790 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.953122 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:14:51.950986 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-serving-cert: secret "maas-api-e2e-ipp-b-535e84-serving-cert" not found Nov 01 04:14:51.953122 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:14:51.951135 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls podName:41574812-6157-45f7-a67b-e952c758491b nodeName:}" failed. No retries permitted until 2026-11-01 04:14:52.451111603 +0000 UTC m=+1904.040206210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls") pod "maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" (UID: "41574812-6157-45f7-a67b-e952c758491b") : secret "maas-api-e2e-ipp-b-535e84-serving-cert" not found Nov 01 04:14:51.953122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.952686 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41574812-6157-45f7-a67b-e952c758491b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:51.972688 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:51.972655 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-metrics-certs\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:52.010440 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:52.010411 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhtl\" (UniqueName: \"kubernetes.io/projected/41574812-6157-45f7-a67b-e952c758491b-kube-api-access-srhtl\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:52.456297 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:52.456264 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:52.458876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:52.458857 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:52.614244 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:52.614212 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:52.984618 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:52.984591 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n"] Nov 01 04:14:52.986516 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:14:52.986490 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41574812_6157_45f7_a67b_e952c758491b.slice/crio-f430fc9064b340eca0369f538e6a7e7283c91b0378916f1c801e7b3a34dcc816 WatchSource:0}: Error finding container f430fc9064b340eca0369f538e6a7e7283c91b0378916f1c801e7b3a34dcc816: Status 404 returned error can't find the container with id f430fc9064b340eca0369f538e6a7e7283c91b0378916f1c801e7b3a34dcc816 Nov 01 04:14:53.014044 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:53.014013 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" event={"ID":"41574812-6157-45f7-a67b-e952c758491b","Type":"ContainerStarted","Data":"f430fc9064b340eca0369f538e6a7e7283c91b0378916f1c801e7b3a34dcc816"} Nov 01 04:14:54.020919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:54.020880 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" event={"ID":"41574812-6157-45f7-a67b-e952c758491b","Type":"ContainerStarted","Data":"bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5"} Nov 01 04:14:54.021349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:54.021112 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:14:54.133800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:54.133701 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" podStartSLOduration=2.30375747 podStartE2EDuration="3.133685648s" totalImagesPullingTime="829.928178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:14:54.131272309 +0000 UTC m=+1905.720366940" watchObservedRunningTime="2026-11-01 04:14:54.133685648 +0000 UTC m=+1905.722780281" Nov 01 04:14:54.686321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:14:54.686284 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:15:00.032359 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.032325 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:15:00.194310 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.194271 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w"] Nov 01 04:15:00.199004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.198980 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.206895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.206847 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4"] Nov 01 04:15:00.213021 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.212998 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.248074 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.248006 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\"" Nov 01 04:15:00.284545 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.284443 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nz6h\" (UniqueName: \"kubernetes.io/projected/c32761b6-ec2b-427b-bd6f-832af4d40e5d-kube-api-access-6nz6h\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.284719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.284555 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32761b6-ec2b-427b-bd6f-832af4d40e5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.284719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.284645 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.284719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.284702 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcxj\" (UniqueName: \"kubernetes.io/projected/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-kube-api-access-8zcxj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.349591 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.349557 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4"] Nov 01 04:15:00.385909 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.385875 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nz6h\" (UniqueName: \"kubernetes.io/projected/c32761b6-ec2b-427b-bd6f-832af4d40e5d-kube-api-access-6nz6h\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.386127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.385925 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32761b6-ec2b-427b-bd6f-832af4d40e5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.386127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.385964 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.386127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.386001 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcxj\" (UniqueName: \"kubernetes.io/projected/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-kube-api-access-8zcxj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.386674 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.386647 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32761b6-ec2b-427b-bd6f-832af4d40e5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.386850 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.386826 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.411157 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.411126 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w"] Nov 01 04:15:00.444331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.444297 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nz6h\" (UniqueName: \"kubernetes.io/projected/c32761b6-ec2b-427b-bd6f-832af4d40e5d-kube-api-access-6nz6h\") pod \"maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.444591 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.444571 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcxj\" (UniqueName: \"kubernetes.io/projected/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-kube-api-access-8zcxj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.517352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.517313 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:00.531159 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.531134 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:00.892961 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.892934 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w"] Nov 01 04:15:00.894684 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:15:00.894659 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32761b6_ec2b_427b_bd6f_832af4d40e5d.slice/crio-aee968a297746cb6713658c856533671ba3289bea4dbd1b2274cfddeeda243d1 WatchSource:0}: Error finding container aee968a297746cb6713658c856533671ba3289bea4dbd1b2274cfddeeda243d1: Status 404 returned error can't find the container with id aee968a297746cb6713658c856533671ba3289bea4dbd1b2274cfddeeda243d1 Nov 01 04:15:00.939396 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:00.939370 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4"] Nov 01 04:15:00.940951 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:15:00.940920 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca9f387_5fb1_465d_bb4f_86a6ae431dd3.slice/crio-6bd149b8c906faaa44f11e67dc7c2171445799ab38327d13e3157b12f5b29659 WatchSource:0}: Error finding container 6bd149b8c906faaa44f11e67dc7c2171445799ab38327d13e3157b12f5b29659: Status 404 returned error can't find the container with id 6bd149b8c906faaa44f11e67dc7c2171445799ab38327d13e3157b12f5b29659 Nov 01 04:15:01.066581 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:01.066545 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" event={"ID":"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3","Type":"ContainerStarted","Data":"6bd149b8c906faaa44f11e67dc7c2171445799ab38327d13e3157b12f5b29659"} Nov 01 04:15:01.067907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:01.067879 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" event={"ID":"c32761b6-ec2b-427b-bd6f-832af4d40e5d","Type":"ContainerStarted","Data":"aee968a297746cb6713658c856533671ba3289bea4dbd1b2274cfddeeda243d1"} Nov 01 04:15:02.075950 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.075860 6735 generic.go:356] "Generic (PLEG): container finished" podID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerID="6af5f97902db3ead6a6bdc96579448b8a33a48d95c4ecae99841bf3655ccf6ae" exitCode=60 Nov 01 04:15:02.076361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.075941 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" event={"ID":"c32761b6-ec2b-427b-bd6f-832af4d40e5d","Type":"ContainerDied","Data":"6af5f97902db3ead6a6bdc96579448b8a33a48d95c4ecae99841bf3655ccf6ae"} Nov 01 04:15:02.076361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.076245 6735 scope.go:122] "RemoveContainer" containerID="6af5f97902db3ead6a6bdc96579448b8a33a48d95c4ecae99841bf3655ccf6ae" Nov 01 04:15:02.077661 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.077640 6735 generic.go:356] "Generic (PLEG): container finished" podID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerID="b445b5d4243a679c3dd0f8b29192f0e32bd098f8ed0ff603501b041d4b13bd99" exitCode=60 Nov 01 04:15:02.077767 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.077723 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" event={"ID":"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3","Type":"ContainerDied","Data":"b445b5d4243a679c3dd0f8b29192f0e32bd098f8ed0ff603501b041d4b13bd99"} Nov 01 04:15:02.078071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.077919 6735 scope.go:122] "RemoveContainer" containerID="b445b5d4243a679c3dd0f8b29192f0e32bd098f8ed0ff603501b041d4b13bd99" Nov 01 04:15:02.184361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.184320 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz"] Nov 01 04:15:02.189845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.189822 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.241753 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.241707 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2d683e-serving-cert\"" Nov 01 04:15:02.245971 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.245941 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz"] Nov 01 04:15:02.312668 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.312635 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fef66ce-be48-4e3a-8770-87c5cee85404-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.312887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.312861 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ggv\" (UniqueName: \"kubernetes.io/projected/4fef66ce-be48-4e3a-8770-87c5cee85404-kube-api-access-p9ggv\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.312967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.312955 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.313116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.313094 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.414005 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.413966 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.414203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.414022 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fef66ce-be48-4e3a-8770-87c5cee85404-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.414203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.414160 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ggv\" (UniqueName: \"kubernetes.io/projected/4fef66ce-be48-4e3a-8770-87c5cee85404-kube-api-access-p9ggv\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.414203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.414196 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.414336 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:02.414300 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-serving-cert: secret "maas-api-e2e-ipp-b-2d683e-serving-cert" not found Nov 01 04:15:02.414377 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:02.414355 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls podName:4fef66ce-be48-4e3a-8770-87c5cee85404 nodeName:}" failed. No retries permitted until 2026-11-01 04:15:02.914337424 +0000 UTC m=+1914.503432032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls") pod "maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" (UID: "4fef66ce-be48-4e3a-8770-87c5cee85404") : secret "maas-api-e2e-ipp-b-2d683e-serving-cert" not found Nov 01 04:15:02.414746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.414723 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fef66ce-be48-4e3a-8770-87c5cee85404-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.416675 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.416658 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.472010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.471981 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ggv\" (UniqueName: \"kubernetes.io/projected/4fef66ce-be48-4e3a-8770-87c5cee85404-kube-api-access-p9ggv\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.936702 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.936667 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:02.939417 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:02.939389 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:03.003411 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:03.003373 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:15:03.085724 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.085688 6735 generic.go:356] "Generic (PLEG): container finished" podID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerID="54dfcba7263fe467e50e9bbdbc05fadd34a428f337921d264d250ab2c022f1fd" exitCode=60 Nov 01 04:15:03.086203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.085772 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" event={"ID":"c32761b6-ec2b-427b-bd6f-832af4d40e5d","Type":"ContainerDied","Data":"54dfcba7263fe467e50e9bbdbc05fadd34a428f337921d264d250ab2c022f1fd"} Nov 01 04:15:03.086203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.085832 6735 scope.go:122] "RemoveContainer" containerID="6af5f97902db3ead6a6bdc96579448b8a33a48d95c4ecae99841bf3655ccf6ae" Nov 01 04:15:03.086203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.086143 6735 scope.go:122] "RemoveContainer" containerID="54dfcba7263fe467e50e9bbdbc05fadd34a428f337921d264d250ab2c022f1fd" Nov 01 04:15:03.086511 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:03.086486 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w_odh-ai-gateway-infra(c32761b6-ec2b-427b-bd6f-832af4d40e5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" Nov 01 04:15:03.087850 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.087829 6735 generic.go:356] "Generic (PLEG): container finished" podID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerID="665fdefc1665c6301d0a6922254863a3939effa239727964adcf5f782d8c8e6e" exitCode=60 Nov 01 04:15:03.087940 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.087860 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" event={"ID":"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3","Type":"ContainerDied","Data":"665fdefc1665c6301d0a6922254863a3939effa239727964adcf5f782d8c8e6e"} Nov 01 04:15:03.088150 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.088136 6735 scope.go:122] "RemoveContainer" containerID="665fdefc1665c6301d0a6922254863a3939effa239727964adcf5f782d8c8e6e" Nov 01 04:15:03.088400 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:03.088384 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4_odh-ai-gateway-infra(4ca9f387-5fb1-465d-bb4f-86a6ae431dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" Nov 01 04:15:03.103465 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.103427 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:03.107132 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.107111 6735 scope.go:122] "RemoveContainer" containerID="b445b5d4243a679c3dd0f8b29192f0e32bd098f8ed0ff603501b041d4b13bd99" Nov 01 04:15:03.478295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:03.478268 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz"] Nov 01 04:15:03.480407 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:15:03.480372 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fef66ce_be48_4e3a_8770_87c5cee85404.slice/crio-cc9cb23181df29ecfc5690633f4b1acc5b9ca2728437f6c703aea5cfaa5fa167 WatchSource:0}: Error finding container cc9cb23181df29ecfc5690633f4b1acc5b9ca2728437f6c703aea5cfaa5fa167: Status 404 returned error can't find the container with id cc9cb23181df29ecfc5690633f4b1acc5b9ca2728437f6c703aea5cfaa5fa167 Nov 01 04:15:04.095885 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:04.095855 6735 scope.go:122] "RemoveContainer" containerID="54dfcba7263fe467e50e9bbdbc05fadd34a428f337921d264d250ab2c022f1fd" Nov 01 04:15:04.096437 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:04.096215 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w_odh-ai-gateway-infra(c32761b6-ec2b-427b-bd6f-832af4d40e5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" Nov 01 04:15:04.097527 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:04.097503 6735 scope.go:122] "RemoveContainer" containerID="665fdefc1665c6301d0a6922254863a3939effa239727964adcf5f782d8c8e6e" Nov 01 04:15:04.097806 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:04.097779 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4_odh-ai-gateway-infra(4ca9f387-5fb1-465d-bb4f-86a6ae431dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" Nov 01 04:15:04.098473 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:04.098452 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" event={"ID":"4fef66ce-be48-4e3a-8770-87c5cee85404","Type":"ContainerStarted","Data":"cc9cb23181df29ecfc5690633f4b1acc5b9ca2728437f6c703aea5cfaa5fa167"} Nov 01 04:15:05.106060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:05.105993 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" event={"ID":"4fef66ce-be48-4e3a-8770-87c5cee85404","Type":"ContainerStarted","Data":"0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8"} Nov 01 04:15:05.106476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:05.106091 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:05.216445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:05.216306 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" podStartSLOduration=2.376671372 podStartE2EDuration="3.216291144s" totalImagesPullingTime="839.619772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:05.215634902 +0000 UTC m=+1916.804729531" watchObservedRunningTime="2026-11-01 04:15:05.216291144 +0000 UTC m=+1916.805385774" Nov 01 04:15:11.121611 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:11.121576 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:15:15.518852 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:15.518822 6735 scope.go:122] "RemoveContainer" containerID="665fdefc1665c6301d0a6922254863a3939effa239727964adcf5f782d8c8e6e" Nov 01 04:15:16.185535 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:16.185443 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" event={"ID":"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3","Type":"ContainerStarted","Data":"280acb0589d5f7275a9b945f7cd5384eda0cf540e3c0a210d0917669c7ecdab8"} Nov 01 04:15:16.294141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:16.294072 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" podStartSLOduration=16.294054504 podStartE2EDuration="16.294054504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:16.293771852 +0000 UTC m=+1927.882866481" watchObservedRunningTime="2026-11-01 04:15:16.294054504 +0000 UTC m=+1927.883149134" Nov 01 04:15:16.520635 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:16.519630 6735 scope.go:122] "RemoveContainer" containerID="54dfcba7263fe467e50e9bbdbc05fadd34a428f337921d264d250ab2c022f1fd" Nov 01 04:15:17.193854 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:17.193766 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" event={"ID":"c32761b6-ec2b-427b-bd6f-832af4d40e5d","Type":"ContainerStarted","Data":"a87e4f892b1a147c18d56ca48118d2cc94d5576f32253d9e28a5d30146d28bb6"} Nov 01 04:15:17.195652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:17.195624 6735 generic.go:356] "Generic (PLEG): container finished" podID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerID="280acb0589d5f7275a9b945f7cd5384eda0cf540e3c0a210d0917669c7ecdab8" exitCode=60 Nov 01 04:15:17.195782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:17.195673 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" event={"ID":"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3","Type":"ContainerDied","Data":"280acb0589d5f7275a9b945f7cd5384eda0cf540e3c0a210d0917669c7ecdab8"} Nov 01 04:15:17.195782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:17.195714 6735 scope.go:122] "RemoveContainer" containerID="665fdefc1665c6301d0a6922254863a3939effa239727964adcf5f782d8c8e6e" Nov 01 04:15:17.196018 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:17.196000 6735 scope.go:122] "RemoveContainer" containerID="280acb0589d5f7275a9b945f7cd5384eda0cf540e3c0a210d0917669c7ecdab8" Nov 01 04:15:17.196317 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:17.196292 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4_odh-ai-gateway-infra(4ca9f387-5fb1-465d-bb4f-86a6ae431dd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" Nov 01 04:15:17.304045 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:17.304006 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4"] Nov 01 04:15:17.304998 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:17.304946 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" podStartSLOduration=17.304929781 podStartE2EDuration="17.304929781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:17.30477839 +0000 UTC m=+1928.893873023" watchObservedRunningTime="2026-11-01 04:15:17.304929781 +0000 UTC m=+1928.894024410" Nov 01 04:15:18.205068 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.205019 6735 generic.go:356] "Generic (PLEG): container finished" podID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerID="a87e4f892b1a147c18d56ca48118d2cc94d5576f32253d9e28a5d30146d28bb6" exitCode=60 Nov 01 04:15:18.205485 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.205068 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" event={"ID":"c32761b6-ec2b-427b-bd6f-832af4d40e5d","Type":"ContainerDied","Data":"a87e4f892b1a147c18d56ca48118d2cc94d5576f32253d9e28a5d30146d28bb6"} Nov 01 04:15:18.205485 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.205159 6735 scope.go:122] "RemoveContainer" containerID="54dfcba7263fe467e50e9bbdbc05fadd34a428f337921d264d250ab2c022f1fd" Nov 01 04:15:18.205626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.205504 6735 scope.go:122] "RemoveContainer" containerID="a87e4f892b1a147c18d56ca48118d2cc94d5576f32253d9e28a5d30146d28bb6" Nov 01 04:15:18.205866 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:18.205840 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w_odh-ai-gateway-infra(c32761b6-ec2b-427b-bd6f-832af4d40e5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" Nov 01 04:15:18.347863 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.347835 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w"] Nov 01 04:15:18.365804 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.365781 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:18.421290 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.421248 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-kube-api-access-8zcxj\" (UniqueName: \"kubernetes.io/projected/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-kube-api-access-8zcxj\") pod \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " Nov 01 04:15:18.421496 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.421451 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-maas-api-service-ca\") pod \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\" (UID: \"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3\") " Nov 01 04:15:18.421818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.421796 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-maas-api-service-ca" pod "4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" (UID: "4ca9f387-5fb1-465d-bb4f-86a6ae431dd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:15:18.421915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.421900 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-maas-api-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:15:18.423509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.423478 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-kube-api-access-8zcxj" pod "4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" (UID: "4ca9f387-5fb1-465d-bb4f-86a6ae431dd3"). InnerVolumeSpecName "kube-api-access-8zcxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:15:18.522702 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:18.522670 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcxj\" (UniqueName: \"kubernetes.io/projected/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3-kube-api-access-8zcxj\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:15:19.215426 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.215404 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" Nov 01 04:15:19.215820 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.215439 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4" event={"ID":"4ca9f387-5fb1-465d-bb4f-86a6ae431dd3","Type":"ContainerDied","Data":"6bd149b8c906faaa44f11e67dc7c2171445799ab38327d13e3157b12f5b29659"} Nov 01 04:15:19.215820 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.215498 6735 scope.go:122] "RemoveContainer" containerID="280acb0589d5f7275a9b945f7cd5384eda0cf540e3c0a210d0917669c7ecdab8" Nov 01 04:15:19.364312 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.364286 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:19.426813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.426784 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4"] Nov 01 04:15:19.431961 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.431934 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32761b6-ec2b-427b-bd6f-832af4d40e5d-kube-api-access-6nz6h\" (UniqueName: \"kubernetes.io/projected/c32761b6-ec2b-427b-bd6f-832af4d40e5d-kube-api-access-6nz6h\") pod \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " Nov 01 04:15:19.432116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.431971 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32761b6-ec2b-427b-bd6f-832af4d40e5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32761b6-ec2b-427b-bd6f-832af4d40e5d-maas-api-service-ca\") pod \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\" (UID: \"c32761b6-ec2b-427b-bd6f-832af4d40e5d\") " Nov 01 04:15:19.432410 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.432381 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32761b6-ec2b-427b-bd6f-832af4d40e5d-maas-api-service-ca" pod "c32761b6-ec2b-427b-bd6f-832af4d40e5d" (UID: "c32761b6-ec2b-427b-bd6f-832af4d40e5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:15:19.433829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.433798 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4adf4c-29891775-h9td4"] Nov 01 04:15:19.434255 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.434233 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32761b6-ec2b-427b-bd6f-832af4d40e5d-kube-api-access-6nz6h" pod "c32761b6-ec2b-427b-bd6f-832af4d40e5d" (UID: "c32761b6-ec2b-427b-bd6f-832af4d40e5d"). InnerVolumeSpecName "kube-api-access-6nz6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:15:19.533543 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.533466 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nz6h\" (UniqueName: \"kubernetes.io/projected/c32761b6-ec2b-427b-bd6f-832af4d40e5d-kube-api-access-6nz6h\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:15:19.533543 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:19.533494 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32761b6-ec2b-427b-bd6f-832af4d40e5d-maas-api-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:15:20.225821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:20.225787 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" Nov 01 04:15:20.226363 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:20.225813 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w" event={"ID":"c32761b6-ec2b-427b-bd6f-832af4d40e5d","Type":"ContainerDied","Data":"aee968a297746cb6713658c856533671ba3289bea4dbd1b2274cfddeeda243d1"} Nov 01 04:15:20.226363 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:20.225893 6735 scope.go:122] "RemoveContainer" containerID="a87e4f892b1a147c18d56ca48118d2cc94d5576f32253d9e28a5d30146d28bb6" Nov 01 04:15:20.380757 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:20.380725 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w"] Nov 01 04:15:20.428334 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:20.428298 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-162191-29891775-gnv9w"] Nov 01 04:15:20.528224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:20.528139 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" path="/var/lib/kubelet/pods/4ca9f387-5fb1-465d-bb4f-86a6ae431dd3/volumes" Nov 01 04:15:20.528656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:20.528634 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" path="/var/lib/kubelet/pods/c32761b6-ec2b-427b-bd6f-832af4d40e5d/volumes" Nov 01 04:15:33.002301 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:15:33.002266 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:15:48.366827 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:48.366790 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:15:51.698645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.698479 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7"] Nov 01 04:15:51.700195 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700166 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:15:51.700360 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700346 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:15:51.700456 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700444 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:15:51.700608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700581 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:15:51.700714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700665 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:15:51.700714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700675 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:15:51.700714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700694 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:15:51.700714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700702 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:15:51.700859 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700836 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:15:51.700859 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700857 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:15:51.700935 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700869 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:15:51.700935 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700882 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:15:51.700935 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.700894 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:15:51.706259 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.706236 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:51.756202 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.756176 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7"] Nov 01 04:15:51.799513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.799482 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3a348f\"" Nov 01 04:15:51.799513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.799486 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3a348f-dockercfg-lpsff\"" Nov 01 04:15:51.799792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.799771 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6f6\" (UniqueName: \"kubernetes.io/projected/2ce26ea6-8551-4917-8742-161d9c8b49d9-kube-api-access-lp6f6\") pod \"payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:51.800048 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.800014 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce26ea6-8551-4917-8742-161d9c8b49d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:51.901804 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.901759 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce26ea6-8551-4917-8742-161d9c8b49d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:51.901990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.901894 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6f6\" (UniqueName: \"kubernetes.io/projected/2ce26ea6-8551-4917-8742-161d9c8b49d9-kube-api-access-lp6f6\") pod \"payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:51.902588 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.902567 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce26ea6-8551-4917-8742-161d9c8b49d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:51.960536 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:51.960498 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6f6\" (UniqueName: \"kubernetes.io/projected/2ce26ea6-8551-4917-8742-161d9c8b49d9-kube-api-access-lp6f6\") pod \"payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:52.023499 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:52.023459 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:52.396675 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:52.396647 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7"] Nov 01 04:15:52.398521 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:15:52.398496 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce26ea6_8551_4917_8742_161d9c8b49d9.slice/crio-a6d6b85fb02f81709c74aeca277498a2ab1db1e79cdbe94ad20f80d519dab740 WatchSource:0}: Error finding container a6d6b85fb02f81709c74aeca277498a2ab1db1e79cdbe94ad20f80d519dab740: Status 404 returned error can't find the container with id a6d6b85fb02f81709c74aeca277498a2ab1db1e79cdbe94ad20f80d519dab740 Nov 01 04:15:52.489128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:52.489086 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" event={"ID":"2ce26ea6-8551-4917-8742-161d9c8b49d9","Type":"ContainerStarted","Data":"fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39"} Nov 01 04:15:52.489269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:52.489254 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:15:52.489319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:52.489286 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" event={"ID":"2ce26ea6-8551-4917-8742-161d9c8b49d9","Type":"ContainerStarted","Data":"a6d6b85fb02f81709c74aeca277498a2ab1db1e79cdbe94ad20f80d519dab740"} Nov 01 04:15:52.651113 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:52.646938 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" podStartSLOduration=1.646917323 podStartE2EDuration="1.646917323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:52.605467836 +0000 UTC m=+1964.194562466" watchObservedRunningTime="2026-11-01 04:15:52.646917323 +0000 UTC m=+1964.236011957" Nov 01 04:15:56.604330 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:15:56.604291 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t"] Nov 01 04:16:01.604625 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:01.604564 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" podUID="4222c967-9ba8-446e-bb84-787570b73e28" containerName="payload-pre-processing" containerID="cri-o://4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12" gracePeriod=25 Nov 01 04:16:01.874883 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:01.874824 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:16:01.942595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:01.942561 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4222c967-9ba8-446e-bb84-787570b73e28-kube-api-access-c2274\" (UniqueName: \"kubernetes.io/projected/4222c967-9ba8-446e-bb84-787570b73e28-kube-api-access-c2274\") pod \"4222c967-9ba8-446e-bb84-787570b73e28\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " Nov 01 04:16:01.942760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:01.942670 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4222c967-9ba8-446e-bb84-787570b73e28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4222c967-9ba8-446e-bb84-787570b73e28-plugins-config-volume\") pod \"4222c967-9ba8-446e-bb84-787570b73e28\" (UID: \"4222c967-9ba8-446e-bb84-787570b73e28\") " Nov 01 04:16:01.943056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:01.943005 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4222c967-9ba8-446e-bb84-787570b73e28-plugins-config-volume" pod "4222c967-9ba8-446e-bb84-787570b73e28" (UID: "4222c967-9ba8-446e-bb84-787570b73e28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:01.944896 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:01.944876 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4222c967-9ba8-446e-bb84-787570b73e28-kube-api-access-c2274" pod "4222c967-9ba8-446e-bb84-787570b73e28" (UID: "4222c967-9ba8-446e-bb84-787570b73e28"). InnerVolumeSpecName "kube-api-access-c2274". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:02.044462 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.044426 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4222c967-9ba8-446e-bb84-787570b73e28-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:02.044462 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.044455 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2274\" (UniqueName: \"kubernetes.io/projected/4222c967-9ba8-446e-bb84-787570b73e28-kube-api-access-c2274\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:02.553850 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.553820 6735 generic.go:356] "Generic (PLEG): container finished" podID="4222c967-9ba8-446e-bb84-787570b73e28" containerID="4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12" exitCode=0 Nov 01 04:16:02.554026 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.553882 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" Nov 01 04:16:02.554026 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.553895 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" event={"ID":"4222c967-9ba8-446e-bb84-787570b73e28","Type":"ContainerDied","Data":"4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12"} Nov 01 04:16:02.554026 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.553947 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t" event={"ID":"4222c967-9ba8-446e-bb84-787570b73e28","Type":"ContainerDied","Data":"286a20780ac4f6073083f314046815aacce71c94dbc12c68472dc76fe5ac5afc"} Nov 01 04:16:02.554026 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.553969 6735 scope.go:122] "RemoveContainer" containerID="4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12" Nov 01 04:16:02.564046 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.564016 6735 scope.go:122] "RemoveContainer" containerID="4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12" Nov 01 04:16:02.564375 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:02.564356 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12\": container with ID starting with 4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12 not found: ID does not exist" containerID="4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12" Nov 01 04:16:02.564457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.564385 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12"} err="failed to get container status \"4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12\": rpc error: code = NotFound desc = could not find container \"4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12\": container with ID starting with 4f7c0b55d1d4de1f03b305809999cf4be3c4682c58c77de573412037797f8e12 not found: ID does not exist" Nov 01 04:16:02.710096 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.710065 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t"] Nov 01 04:16:02.756578 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:02.756545 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-7cccb557f5-hw97t"] Nov 01 04:16:03.001704 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:03.001672 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:16:03.498684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:03.498658 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:16:04.525870 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:04.525839 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4222c967-9ba8-446e-bb84-787570b73e28" path="/var/lib/kubelet/pods/4222c967-9ba8-446e-bb84-787570b73e28/volumes" Nov 01 04:16:15.503396 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:15.503320 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz"] Nov 01 04:16:17.512088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:17.512051 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh"] Nov 01 04:16:17.610463 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:17.610431 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p"] Nov 01 04:16:22.513112 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.513026 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" podUID="c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" containerName="payload-processing" containerID="cri-o://e2d2c6b7fe878dcf9ecf0b73652814c1d2ee23da0ad8ab158602ac9efa5b6b20" gracePeriod=25 Nov 01 04:16:22.610783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.610710 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" podUID="de3edd7b-9086-42b8-a4ad-c9575db43548" containerName="payload-pre-processing" containerID="cri-o://1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d" gracePeriod=25 Nov 01 04:16:22.695378 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.695345 6735 generic.go:356] "Generic (PLEG): container finished" podID="c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" containerID="e2d2c6b7fe878dcf9ecf0b73652814c1d2ee23da0ad8ab158602ac9efa5b6b20" exitCode=0 Nov 01 04:16:22.695554 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.695515 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" event={"ID":"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba","Type":"ContainerDied","Data":"e2d2c6b7fe878dcf9ecf0b73652814c1d2ee23da0ad8ab158602ac9efa5b6b20"} Nov 01 04:16:22.807481 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.807452 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:16:22.875361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.875338 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:16:22.928447 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.928413 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-kube-api-access-lvd9q\" (UniqueName: \"kubernetes.io/projected/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-kube-api-access-lvd9q\") pod \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " Nov 01 04:16:22.928634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.928482 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3edd7b-9086-42b8-a4ad-c9575db43548-kube-api-access-t7ls8\" (UniqueName: \"kubernetes.io/projected/de3edd7b-9086-42b8-a4ad-c9575db43548-kube-api-access-t7ls8\") pod \"de3edd7b-9086-42b8-a4ad-c9575db43548\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " Nov 01 04:16:22.928939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.928914 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-plugins-config-volume\") pod \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\" (UID: \"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba\") " Nov 01 04:16:22.929111 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.929085 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3edd7b-9086-42b8-a4ad-c9575db43548-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3edd7b-9086-42b8-a4ad-c9575db43548-plugins-config-volume\") pod \"de3edd7b-9086-42b8-a4ad-c9575db43548\" (UID: \"de3edd7b-9086-42b8-a4ad-c9575db43548\") " Nov 01 04:16:22.929649 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.929383 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-plugins-config-volume" pod "c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" (UID: "c4dd5772-0c1e-40e8-8709-d3b8ec86ddba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:22.929649 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.929603 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3edd7b-9086-42b8-a4ad-c9575db43548-plugins-config-volume" pod "de3edd7b-9086-42b8-a4ad-c9575db43548" (UID: "de3edd7b-9086-42b8-a4ad-c9575db43548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:22.929816 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.929744 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:22.929816 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.929765 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3edd7b-9086-42b8-a4ad-c9575db43548-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:22.931350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.931327 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-kube-api-access-lvd9q" pod "c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" (UID: "c4dd5772-0c1e-40e8-8709-d3b8ec86ddba"). InnerVolumeSpecName "kube-api-access-lvd9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:22.931427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:22.931347 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3edd7b-9086-42b8-a4ad-c9575db43548-kube-api-access-t7ls8" pod "de3edd7b-9086-42b8-a4ad-c9575db43548" (UID: "de3edd7b-9086-42b8-a4ad-c9575db43548"). InnerVolumeSpecName "kube-api-access-t7ls8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:23.031573 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.031498 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvd9q\" (UniqueName: \"kubernetes.io/projected/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba-kube-api-access-lvd9q\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:23.031573 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.031528 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ls8\" (UniqueName: \"kubernetes.io/projected/de3edd7b-9086-42b8-a4ad-c9575db43548-kube-api-access-t7ls8\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:23.703483 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.703449 6735 generic.go:356] "Generic (PLEG): container finished" podID="de3edd7b-9086-42b8-a4ad-c9575db43548" containerID="1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d" exitCode=0 Nov 01 04:16:23.703949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.703494 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" Nov 01 04:16:23.703949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.703528 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" event={"ID":"de3edd7b-9086-42b8-a4ad-c9575db43548","Type":"ContainerDied","Data":"1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d"} Nov 01 04:16:23.703949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.703583 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p" event={"ID":"de3edd7b-9086-42b8-a4ad-c9575db43548","Type":"ContainerDied","Data":"eda8a898a7b3289335a36903c771579de295cbc0465757f4c1da5061f3c45781"} Nov 01 04:16:23.703949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.703604 6735 scope.go:122] "RemoveContainer" containerID="1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d" Nov 01 04:16:23.705086 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.704986 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" event={"ID":"c4dd5772-0c1e-40e8-8709-d3b8ec86ddba","Type":"ContainerDied","Data":"2c71a52d0218ab2b5b6cff0254aba8e0facd1a4d1282f31f57bbc0341c80f0b8"} Nov 01 04:16:23.705086 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.704995 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh" Nov 01 04:16:23.715606 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.715585 6735 scope.go:122] "RemoveContainer" containerID="1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d" Nov 01 04:16:23.715868 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:23.715850 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d\": container with ID starting with 1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d not found: ID does not exist" containerID="1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d" Nov 01 04:16:23.715948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.715879 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d"} err="failed to get container status \"1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d\": rpc error: code = NotFound desc = could not find container \"1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d\": container with ID starting with 1d3b6ebaaa716990564f71ca01d1ea953fa836793bc16a91b0bc3fcb6c9c783d not found: ID does not exist" Nov 01 04:16:23.715948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.715903 6735 scope.go:122] "RemoveContainer" containerID="e2d2c6b7fe878dcf9ecf0b73652814c1d2ee23da0ad8ab158602ac9efa5b6b20" Nov 01 04:16:23.858253 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.858172 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p"] Nov 01 04:16:23.905691 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.905654 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-535e84-747749d5bc-vjz7p"] Nov 01 04:16:23.967058 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.967001 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh"] Nov 01 04:16:23.969476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:23.969450 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-535e84-557577dbbc-xrxhh"] Nov 01 04:16:24.526120 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:24.526090 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" path="/var/lib/kubelet/pods/c4dd5772-0c1e-40e8-8709-d3b8ec86ddba/volumes" Nov 01 04:16:24.526548 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:24.526532 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3edd7b-9086-42b8-a4ad-c9575db43548" path="/var/lib/kubelet/pods/de3edd7b-9086-42b8-a4ad-c9575db43548/volumes" Nov 01 04:16:26.484479 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:26.484450 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68"] Nov 01 04:16:26.531797 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:26.531763 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw"] Nov 01 04:16:27.426212 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:27.426177 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:16:30.528843 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.528802 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" podUID="8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" containerName="main" containerID="cri-o://73a91d48e0be64499b5278debb74549cc035d0336ca25f8d95eb63e71598c702" gracePeriod=45 Nov 01 04:16:30.752665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.752635 6735 generic.go:356] "Generic (PLEG): container finished" podID="8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" containerID="73a91d48e0be64499b5278debb74549cc035d0336ca25f8d95eb63e71598c702" exitCode=0 Nov 01 04:16:30.752839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.752711 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" event={"ID":"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3","Type":"ContainerDied","Data":"73a91d48e0be64499b5278debb74549cc035d0336ca25f8d95eb63e71598c702"} Nov 01 04:16:30.811019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.810995 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:16:30.942754 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.942718 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-dshm\") pod \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " Nov 01 04:16:30.942974 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.942949 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-kube-api-access-zg9fq\" (UniqueName: \"kubernetes.io/projected/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-kube-api-access-zg9fq\") pod \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " Nov 01 04:16:30.943074 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.942995 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tmp-dir\") pod \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " Nov 01 04:16:30.943358 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.943325 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tmp-dir" pod "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" (UID: "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:16:30.943358 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.943337 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-home" pod "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" (UID: "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:16:30.943683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.943377 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-home\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-home\") pod \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " Nov 01 04:16:30.943683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.943450 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-model-cache\") pod \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " Nov 01 04:16:30.943683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.943496 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tls-certs\") pod \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\" (UID: \"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3\") " Nov 01 04:16:30.944654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.944626 6735 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tmp-dir\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:30.944821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.944792 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-model-cache" pod "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" (UID: "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:16:30.956108 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.945565 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-dshm" pod "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" (UID: "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:16:30.956108 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.944807 6735 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-home\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:30.966569 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.966537 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-kube-api-access-zg9fq" pod "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" (UID: "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3"). InnerVolumeSpecName "kube-api-access-zg9fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:30.966677 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:30.966597 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tls-certs" pod "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" (UID: "8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:16:31.057853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.057775 6735 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-model-cache\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:31.057853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.057805 6735 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-tls-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:31.057853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.057814 6735 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-dshm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:31.057853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.057843 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9fq\" (UniqueName: \"kubernetes.io/projected/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3-kube-api-access-zg9fq\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:31.484780 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.484720 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" podUID="895cdd1f-c503-41e1-a4e6-383be3fb695a" containerName="payload-processing" containerID="cri-o://d1ea50ee477f37d009abae6a6ae68624960eba2eb910fd5968b4b8e5205a6224" gracePeriod=25 Nov 01 04:16:31.532793 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.532749 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" podUID="5c0775ed-89fd-4eb8-86ae-3338a74051a2" containerName="payload-pre-processing" containerID="cri-o://43fda6e2bda2b5401ebf3b5fc20798e9f853e91e03fc49d2fcfadd87a422bdbe" gracePeriod=25 Nov 01 04:16:31.762482 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.762442 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" event={"ID":"8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3","Type":"ContainerDied","Data":"932056b931d2d47c4e89da3fd1400158e54913247e34441929c4c8e9cf74f4a4"} Nov 01 04:16:31.762638 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.762509 6735 scope.go:122] "RemoveContainer" containerID="73a91d48e0be64499b5278debb74549cc035d0336ca25f8d95eb63e71598c702" Nov 01 04:16:31.762940 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.762911 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz" Nov 01 04:16:31.795798 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.795560 6735 generic.go:356] "Generic (PLEG): container finished" podID="895cdd1f-c503-41e1-a4e6-383be3fb695a" containerID="d1ea50ee477f37d009abae6a6ae68624960eba2eb910fd5968b4b8e5205a6224" exitCode=0 Nov 01 04:16:31.795798 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.795687 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" event={"ID":"895cdd1f-c503-41e1-a4e6-383be3fb695a","Type":"ContainerDied","Data":"d1ea50ee477f37d009abae6a6ae68624960eba2eb910fd5968b4b8e5205a6224"} Nov 01 04:16:31.798449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.798425 6735 generic.go:356] "Generic (PLEG): container finished" podID="5c0775ed-89fd-4eb8-86ae-3338a74051a2" containerID="43fda6e2bda2b5401ebf3b5fc20798e9f853e91e03fc49d2fcfadd87a422bdbe" exitCode=0 Nov 01 04:16:31.798743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.798628 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" event={"ID":"5c0775ed-89fd-4eb8-86ae-3338a74051a2","Type":"ContainerDied","Data":"43fda6e2bda2b5401ebf3b5fc20798e9f853e91e03fc49d2fcfadd87a422bdbe"} Nov 01 04:16:31.857776 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.857743 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:16:31.859968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.859950 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:16:31.926699 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.926669 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz"] Nov 01 04:16:31.972832 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.972806 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fb1c1f/tenantcfg-model-fb1c1f-kserve-648bdf77c-gjmhz"] Nov 01 04:16:31.978685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.978663 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895cdd1f-c503-41e1-a4e6-383be3fb695a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895cdd1f-c503-41e1-a4e6-383be3fb695a-plugins-config-volume\") pod \"895cdd1f-c503-41e1-a4e6-383be3fb695a\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " Nov 01 04:16:31.978813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.978731 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0775ed-89fd-4eb8-86ae-3338a74051a2-kube-api-access-4lg4k\" (UniqueName: \"kubernetes.io/projected/5c0775ed-89fd-4eb8-86ae-3338a74051a2-kube-api-access-4lg4k\") pod \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " Nov 01 04:16:31.978857 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.978846 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0775ed-89fd-4eb8-86ae-3338a74051a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0775ed-89fd-4eb8-86ae-3338a74051a2-plugins-config-volume\") pod \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\" (UID: \"5c0775ed-89fd-4eb8-86ae-3338a74051a2\") " Nov 01 04:16:31.978903 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.978894 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895cdd1f-c503-41e1-a4e6-383be3fb695a-kube-api-access-bss6j\" (UniqueName: \"kubernetes.io/projected/895cdd1f-c503-41e1-a4e6-383be3fb695a-kube-api-access-bss6j\") pod \"895cdd1f-c503-41e1-a4e6-383be3fb695a\" (UID: \"895cdd1f-c503-41e1-a4e6-383be3fb695a\") " Nov 01 04:16:31.979084 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.979064 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895cdd1f-c503-41e1-a4e6-383be3fb695a-plugins-config-volume" pod "895cdd1f-c503-41e1-a4e6-383be3fb695a" (UID: "895cdd1f-c503-41e1-a4e6-383be3fb695a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:31.979310 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.979288 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0775ed-89fd-4eb8-86ae-3338a74051a2-plugins-config-volume" pod "5c0775ed-89fd-4eb8-86ae-3338a74051a2" (UID: "5c0775ed-89fd-4eb8-86ae-3338a74051a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:31.979411 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.979393 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895cdd1f-c503-41e1-a4e6-383be3fb695a-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:31.979448 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.979420 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0775ed-89fd-4eb8-86ae-3338a74051a2-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:31.981385 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.981364 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895cdd1f-c503-41e1-a4e6-383be3fb695a-kube-api-access-bss6j" pod "895cdd1f-c503-41e1-a4e6-383be3fb695a" (UID: "895cdd1f-c503-41e1-a4e6-383be3fb695a"). InnerVolumeSpecName "kube-api-access-bss6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:31.981385 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:31.981380 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0775ed-89fd-4eb8-86ae-3338a74051a2-kube-api-access-4lg4k" pod "5c0775ed-89fd-4eb8-86ae-3338a74051a2" (UID: "5c0775ed-89fd-4eb8-86ae-3338a74051a2"). InnerVolumeSpecName "kube-api-access-4lg4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:32.080516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.080439 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lg4k\" (UniqueName: \"kubernetes.io/projected/5c0775ed-89fd-4eb8-86ae-3338a74051a2-kube-api-access-4lg4k\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:32.080516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.080467 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bss6j\" (UniqueName: \"kubernetes.io/projected/895cdd1f-c503-41e1-a4e6-383be3fb695a-kube-api-access-bss6j\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:32.530297 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.530261 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" path="/var/lib/kubelet/pods/8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3/volumes" Nov 01 04:16:32.805981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.805894 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" Nov 01 04:16:32.805981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.805934 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw" event={"ID":"5c0775ed-89fd-4eb8-86ae-3338a74051a2","Type":"ContainerDied","Data":"726909b376b53b9f6f02288cf9b59f6e1e8d459ed36727d1131ce6d5f29784f0"} Nov 01 04:16:32.806526 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.805993 6735 scope.go:122] "RemoveContainer" containerID="43fda6e2bda2b5401ebf3b5fc20798e9f853e91e03fc49d2fcfadd87a422bdbe" Nov 01 04:16:32.808697 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.808675 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" event={"ID":"895cdd1f-c503-41e1-a4e6-383be3fb695a","Type":"ContainerDied","Data":"dce15c1c381279788c344880e2c1496aaef0222234c04196547ba9939ddd1a84"} Nov 01 04:16:32.808830 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.808763 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68" Nov 01 04:16:32.815879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.815863 6735 scope.go:122] "RemoveContainer" containerID="d1ea50ee477f37d009abae6a6ae68624960eba2eb910fd5968b4b8e5205a6224" Nov 01 04:16:32.960439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:32.960402 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw"] Nov 01 04:16:33.001913 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:33.001881 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:16:33.006868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:33.006843 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d683e-554b76f66c-9jxvw"] Nov 01 04:16:33.067845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:33.067770 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68"] Nov 01 04:16:33.071288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:33.071266 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d683e-65cd844475-hps68"] Nov 01 04:16:34.525888 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:34.525854 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0775ed-89fd-4eb8-86ae-3338a74051a2" path="/var/lib/kubelet/pods/5c0775ed-89fd-4eb8-86ae-3338a74051a2/volumes" Nov 01 04:16:34.526305 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:34.526287 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895cdd1f-c503-41e1-a4e6-383be3fb695a" path="/var/lib/kubelet/pods/895cdd1f-c503-41e1-a4e6-383be3fb695a/volumes" Nov 01 04:16:39.937577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:39.937544 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg"] Nov 01 04:16:39.937970 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:39.937759 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" podUID="e59f4b35-8965-419a-a5e1-938ee6b4f31f" containerName="maas-api" containerID="cri-o://170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2" gracePeriod=30 Nov 01 04:16:40.249550 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.249525 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:16:40.291198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.291162 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls\") pod \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " Nov 01 04:16:40.291348 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.291317 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59f4b35-8965-419a-a5e1-938ee6b4f31f-kube-api-access-dzdfc\" (UniqueName: \"kubernetes.io/projected/e59f4b35-8965-419a-a5e1-938ee6b4f31f-kube-api-access-dzdfc\") pod \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " Nov 01 04:16:40.291425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.291348 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-metrics-certs\") pod \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " Nov 01 04:16:40.291500 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.291486 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59f4b35-8965-419a-a5e1-938ee6b4f31f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f4b35-8965-419a-a5e1-938ee6b4f31f-openshift-service-ca\") pod \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\" (UID: \"e59f4b35-8965-419a-a5e1-938ee6b4f31f\") " Nov 01 04:16:40.291942 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.291854 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59f4b35-8965-419a-a5e1-938ee6b4f31f-openshift-service-ca" pod "e59f4b35-8965-419a-a5e1-938ee6b4f31f" (UID: "e59f4b35-8965-419a-a5e1-938ee6b4f31f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:40.293493 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.293470 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls" pod "e59f4b35-8965-419a-a5e1-938ee6b4f31f" (UID: "e59f4b35-8965-419a-a5e1-938ee6b4f31f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:16:40.293599 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.293488 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-metrics-certs" pod "e59f4b35-8965-419a-a5e1-938ee6b4f31f" (UID: "e59f4b35-8965-419a-a5e1-938ee6b4f31f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:16:40.293599 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.293470 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59f4b35-8965-419a-a5e1-938ee6b4f31f-kube-api-access-dzdfc" pod "e59f4b35-8965-419a-a5e1-938ee6b4f31f" (UID: "e59f4b35-8965-419a-a5e1-938ee6b4f31f"). InnerVolumeSpecName "kube-api-access-dzdfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:40.394553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.394516 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:40.394797 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.394770 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdfc\" (UniqueName: \"kubernetes.io/projected/e59f4b35-8965-419a-a5e1-938ee6b4f31f-kube-api-access-dzdfc\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:40.394797 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.394800 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e59f4b35-8965-419a-a5e1-938ee6b4f31f-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:40.394983 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.394815 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59f4b35-8965-419a-a5e1-938ee6b4f31f-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:40.404450 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.404415 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9"] Nov 01 04:16:40.404712 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.404685 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" podUID="956fe986-cddf-4077-b9df-4a4a5b93a1d3" containerName="payload-processing" containerID="cri-o://4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612" gracePeriod=30 Nov 01 04:16:40.459117 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.459087 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx"] Nov 01 04:16:40.459319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.459295 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" podUID="b102f43a-f1e6-4295-a344-f336c6e54c5b" containerName="payload-pre-processing" containerID="cri-o://6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8" gracePeriod=30 Nov 01 04:16:40.719973 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.719943 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:16:40.749432 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.749403 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:16:40.799186 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799114 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b102f43a-f1e6-4295-a344-f336c6e54c5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b102f43a-f1e6-4295-a344-f336c6e54c5b-plugins-config-volume\") pod \"b102f43a-f1e6-4295-a344-f336c6e54c5b\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " Nov 01 04:16:40.799310 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799195 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956fe986-cddf-4077-b9df-4a4a5b93a1d3-kube-api-access-rfk2g\" (UniqueName: \"kubernetes.io/projected/956fe986-cddf-4077-b9df-4a4a5b93a1d3-kube-api-access-rfk2g\") pod \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " Nov 01 04:16:40.799369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799300 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b102f43a-f1e6-4295-a344-f336c6e54c5b-kube-api-access-djrj5\" (UniqueName: \"kubernetes.io/projected/b102f43a-f1e6-4295-a344-f336c6e54c5b-kube-api-access-djrj5\") pod \"b102f43a-f1e6-4295-a344-f336c6e54c5b\" (UID: \"b102f43a-f1e6-4295-a344-f336c6e54c5b\") " Nov 01 04:16:40.799423 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799364 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956fe986-cddf-4077-b9df-4a4a5b93a1d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956fe986-cddf-4077-b9df-4a4a5b93a1d3-plugins-config-volume\") pod \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\" (UID: \"956fe986-cddf-4077-b9df-4a4a5b93a1d3\") " Nov 01 04:16:40.799605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799572 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b102f43a-f1e6-4295-a344-f336c6e54c5b-plugins-config-volume" pod "b102f43a-f1e6-4295-a344-f336c6e54c5b" (UID: "b102f43a-f1e6-4295-a344-f336c6e54c5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:40.799760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799735 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956fe986-cddf-4077-b9df-4a4a5b93a1d3-plugins-config-volume" pod "956fe986-cddf-4077-b9df-4a4a5b93a1d3" (UID: "956fe986-cddf-4077-b9df-4a4a5b93a1d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:40.799970 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799947 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956fe986-cddf-4077-b9df-4a4a5b93a1d3-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:40.800098 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.799973 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b102f43a-f1e6-4295-a344-f336c6e54c5b-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:40.801683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.801664 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956fe986-cddf-4077-b9df-4a4a5b93a1d3-kube-api-access-rfk2g" pod "956fe986-cddf-4077-b9df-4a4a5b93a1d3" (UID: "956fe986-cddf-4077-b9df-4a4a5b93a1d3"). InnerVolumeSpecName "kube-api-access-rfk2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:40.801910 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.801891 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b102f43a-f1e6-4295-a344-f336c6e54c5b-kube-api-access-djrj5" pod "b102f43a-f1e6-4295-a344-f336c6e54c5b" (UID: "b102f43a-f1e6-4295-a344-f336c6e54c5b"). InnerVolumeSpecName "kube-api-access-djrj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:40.863284 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.863251 6735 generic.go:356] "Generic (PLEG): container finished" podID="e59f4b35-8965-419a-a5e1-938ee6b4f31f" containerID="170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2" exitCode=0 Nov 01 04:16:40.863474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.863295 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" Nov 01 04:16:40.863474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.863333 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" event={"ID":"e59f4b35-8965-419a-a5e1-938ee6b4f31f","Type":"ContainerDied","Data":"170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2"} Nov 01 04:16:40.863474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.863385 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg" event={"ID":"e59f4b35-8965-419a-a5e1-938ee6b4f31f","Type":"ContainerDied","Data":"a7a43692681b4d99936afc61d160c04b0c8d782b4a1bf2f5e008157681203b26"} Nov 01 04:16:40.863474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.863417 6735 scope.go:122] "RemoveContainer" containerID="170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2" Nov 01 04:16:40.865319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.865290 6735 generic.go:356] "Generic (PLEG): container finished" podID="b102f43a-f1e6-4295-a344-f336c6e54c5b" containerID="6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8" exitCode=0 Nov 01 04:16:40.865486 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.865471 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" Nov 01 04:16:40.866520 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.866477 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" event={"ID":"b102f43a-f1e6-4295-a344-f336c6e54c5b","Type":"ContainerDied","Data":"6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8"} Nov 01 04:16:40.866631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.866534 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx" event={"ID":"b102f43a-f1e6-4295-a344-f336c6e54c5b","Type":"ContainerDied","Data":"e967d9f6e577eee75e8e0941f798028ce16d0208ff30f4b7f79bd273af860f20"} Nov 01 04:16:40.867892 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.867872 6735 generic.go:356] "Generic (PLEG): container finished" podID="956fe986-cddf-4077-b9df-4a4a5b93a1d3" containerID="4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612" exitCode=0 Nov 01 04:16:40.868023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.867957 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" event={"ID":"956fe986-cddf-4077-b9df-4a4a5b93a1d3","Type":"ContainerDied","Data":"4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612"} Nov 01 04:16:40.868023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.867994 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" event={"ID":"956fe986-cddf-4077-b9df-4a4a5b93a1d3","Type":"ContainerDied","Data":"4e10293fac6367fd3a81f3fb46bacdbff7df1870281174c986baaebbc38a5e46"} Nov 01 04:16:40.868189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.868073 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9" Nov 01 04:16:40.876646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.876621 6735 scope.go:122] "RemoveContainer" containerID="170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2" Nov 01 04:16:40.876951 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:40.876928 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2\": container with ID starting with 170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2 not found: ID does not exist" containerID="170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2" Nov 01 04:16:40.877093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.876961 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2"} err="failed to get container status \"170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2\": rpc error: code = NotFound desc = could not find container \"170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2\": container with ID starting with 170ffe449173b8b7555ece1cd60ab8868babb826199ca1a9587e84b2247961e2 not found: ID does not exist" Nov 01 04:16:40.877093 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.876984 6735 scope.go:122] "RemoveContainer" containerID="6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8" Nov 01 04:16:40.886964 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.886943 6735 scope.go:122] "RemoveContainer" containerID="6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8" Nov 01 04:16:40.887226 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:40.887206 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8\": container with ID starting with 6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8 not found: ID does not exist" containerID="6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8" Nov 01 04:16:40.887297 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.887236 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8"} err="failed to get container status \"6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8\": rpc error: code = NotFound desc = could not find container \"6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8\": container with ID starting with 6ffb019ccf581a15825e8fe7d4c09130e5a029816a4ccfb3ad8d916294ba0db8 not found: ID does not exist" Nov 01 04:16:40.887297 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.887258 6735 scope.go:122] "RemoveContainer" containerID="4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612" Nov 01 04:16:40.895403 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.895385 6735 scope.go:122] "RemoveContainer" containerID="4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612" Nov 01 04:16:40.895672 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:40.895655 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612\": container with ID starting with 4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612 not found: ID does not exist" containerID="4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612" Nov 01 04:16:40.895734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.895684 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612"} err="failed to get container status \"4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612\": rpc error: code = NotFound desc = could not find container \"4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612\": container with ID starting with 4255dbd564f218b475468029d49c03bbe5841002f471f5bfd45fe5275605c612 not found: ID does not exist" Nov 01 04:16:40.900564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.900546 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfk2g\" (UniqueName: \"kubernetes.io/projected/956fe986-cddf-4077-b9df-4a4a5b93a1d3-kube-api-access-rfk2g\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:40.900621 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:40.900567 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djrj5\" (UniqueName: \"kubernetes.io/projected/b102f43a-f1e6-4295-a344-f336c6e54c5b-kube-api-access-djrj5\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:41.073390 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:41.073308 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg"] Nov 01 04:16:41.077288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:41.077264 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-157fed-5fc7987d66-tt8mg"] Nov 01 04:16:41.180144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:41.180103 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9"] Nov 01 04:16:41.186158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:41.186124 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-157fed-86b5f678fd-tkcn9"] Nov 01 04:16:41.288483 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:41.288446 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx"] Nov 01 04:16:41.293416 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:41.293388 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-157fed-65fb76cb59-4jzbx"] Nov 01 04:16:42.526893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:42.526862 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956fe986-cddf-4077-b9df-4a4a5b93a1d3" path="/var/lib/kubelet/pods/956fe986-cddf-4077-b9df-4a4a5b93a1d3/volumes" Nov 01 04:16:42.527455 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:42.527433 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b102f43a-f1e6-4295-a344-f336c6e54c5b" path="/var/lib/kubelet/pods/b102f43a-f1e6-4295-a344-f336c6e54c5b/volumes" Nov 01 04:16:42.527974 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:42.527950 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59f4b35-8965-419a-a5e1-938ee6b4f31f" path="/var/lib/kubelet/pods/e59f4b35-8965-419a-a5e1-938ee6b4f31f/volumes" Nov 01 04:16:46.454331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.454293 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh"] Nov 01 04:16:46.454812 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.454578 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" podUID="912227d8-0b7e-47c0-9585-46e708095369" containerName="maas-api" containerID="cri-o://92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738" gracePeriod=30 Nov 01 04:16:46.755178 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.755152 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:16:46.817654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.817624 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls\") pod \"912227d8-0b7e-47c0-9585-46e708095369\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " Nov 01 04:16:46.817829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.817665 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912227d8-0b7e-47c0-9585-46e708095369-kube-api-access-s4sjv\" (UniqueName: \"kubernetes.io/projected/912227d8-0b7e-47c0-9585-46e708095369-kube-api-access-s4sjv\") pod \"912227d8-0b7e-47c0-9585-46e708095369\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " Nov 01 04:16:46.817829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.817698 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-metrics-certs\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-metrics-certs\") pod \"912227d8-0b7e-47c0-9585-46e708095369\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " Nov 01 04:16:46.817968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.817859 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912227d8-0b7e-47c0-9585-46e708095369-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912227d8-0b7e-47c0-9585-46e708095369-openshift-service-ca\") pod \"912227d8-0b7e-47c0-9585-46e708095369\" (UID: \"912227d8-0b7e-47c0-9585-46e708095369\") " Nov 01 04:16:46.818282 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.818252 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912227d8-0b7e-47c0-9585-46e708095369-openshift-service-ca" pod "912227d8-0b7e-47c0-9585-46e708095369" (UID: "912227d8-0b7e-47c0-9585-46e708095369"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:16:46.818721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.818699 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912227d8-0b7e-47c0-9585-46e708095369-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:46.820020 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.819997 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-metrics-certs" pod "912227d8-0b7e-47c0-9585-46e708095369" (UID: "912227d8-0b7e-47c0-9585-46e708095369"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:16:46.820133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.820000 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912227d8-0b7e-47c0-9585-46e708095369-kube-api-access-s4sjv" pod "912227d8-0b7e-47c0-9585-46e708095369" (UID: "912227d8-0b7e-47c0-9585-46e708095369"). InnerVolumeSpecName "kube-api-access-s4sjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:16:46.820133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.820071 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls" pod "912227d8-0b7e-47c0-9585-46e708095369" (UID: "912227d8-0b7e-47c0-9585-46e708095369"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:16:46.919818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.919782 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:46.919818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.919819 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4sjv\" (UniqueName: \"kubernetes.io/projected/912227d8-0b7e-47c0-9585-46e708095369-kube-api-access-s4sjv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:46.920170 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.919835 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912227d8-0b7e-47c0-9585-46e708095369-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:16:46.923915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.923878 6735 generic.go:356] "Generic (PLEG): container finished" podID="912227d8-0b7e-47c0-9585-46e708095369" containerID="92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738" exitCode=0 Nov 01 04:16:46.924144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.924124 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" Nov 01 04:16:46.924409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.924149 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" event={"ID":"912227d8-0b7e-47c0-9585-46e708095369","Type":"ContainerDied","Data":"92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738"} Nov 01 04:16:46.924494 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.924452 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh" event={"ID":"912227d8-0b7e-47c0-9585-46e708095369","Type":"ContainerDied","Data":"2aa8c17a4683b0042f1536da5bb63d2535f835e3183c9bd922239f370393f7fc"} Nov 01 04:16:46.924562 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.924497 6735 scope.go:122] "RemoveContainer" containerID="92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738" Nov 01 04:16:46.939130 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.939108 6735 scope.go:122] "RemoveContainer" containerID="92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738" Nov 01 04:16:46.939443 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:46.939419 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738\": container with ID starting with 92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738 not found: ID does not exist" containerID="92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738" Nov 01 04:16:46.939503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:46.939457 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738"} err="failed to get container status \"92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738\": rpc error: code = NotFound desc = could not find container \"92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738\": container with ID starting with 92f14800bf407468c39a5ae4d4521c2ac0d5ed4d67ed099ec1e322662b748738 not found: ID does not exist" Nov 01 04:16:47.081433 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:47.081317 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh"] Nov 01 04:16:47.088961 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:47.088926 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5b71cd-7897fcb4bd-6vnjh"] Nov 01 04:16:48.113982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.113946 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd"] Nov 01 04:16:48.114514 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114500 6735 cpu_manager.go:414] "containerMap: removing container" podUID="b102f43a-f1e6-4295-a344-f336c6e54c5b" containerName="payload-pre-processing" Nov 01 04:16:48.114559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114516 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b102f43a-f1e6-4295-a344-f336c6e54c5b" containerName="payload-pre-processing" Nov 01 04:16:48.114559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114530 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:16:48.114559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114537 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32761b6-ec2b-427b-bd6f-832af4d40e5d" containerName="cleanup" Nov 01 04:16:48.114559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114550 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e59f4b35-8965-419a-a5e1-938ee6b4f31f" containerName="maas-api" Nov 01 04:16:48.114559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114554 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59f4b35-8965-419a-a5e1-938ee6b4f31f" containerName="maas-api" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114563 6735 cpu_manager.go:414] "containerMap: removing container" podUID="de3edd7b-9086-42b8-a4ad-c9575db43548" containerName="payload-pre-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114567 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3edd7b-9086-42b8-a4ad-c9575db43548" containerName="payload-pre-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114574 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" containerName="payload-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114579 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" containerName="payload-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114588 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114593 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114600 6735 cpu_manager.go:414] "containerMap: removing container" podUID="8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" containerName="main" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114604 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" containerName="main" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114634 6735 cpu_manager.go:414] "containerMap: removing container" podUID="895cdd1f-c503-41e1-a4e6-383be3fb695a" containerName="payload-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114640 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895cdd1f-c503-41e1-a4e6-383be3fb695a" containerName="payload-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114647 6735 cpu_manager.go:414] "containerMap: removing container" podUID="912227d8-0b7e-47c0-9585-46e708095369" containerName="maas-api" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114652 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912227d8-0b7e-47c0-9585-46e708095369" containerName="maas-api" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114660 6735 cpu_manager.go:414] "containerMap: removing container" podUID="956fe986-cddf-4077-b9df-4a4a5b93a1d3" containerName="payload-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114664 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956fe986-cddf-4077-b9df-4a4a5b93a1d3" containerName="payload-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114680 6735 cpu_manager.go:414] "containerMap: removing container" podUID="5c0775ed-89fd-4eb8-86ae-3338a74051a2" containerName="payload-pre-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114685 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0775ed-89fd-4eb8-86ae-3338a74051a2" containerName="payload-pre-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114693 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4222c967-9ba8-446e-bb84-787570b73e28" containerName="payload-pre-processing" Nov 01 04:16:48.114706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114699 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4222c967-9ba8-446e-bb84-787570b73e28" containerName="payload-pre-processing" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114809 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="b102f43a-f1e6-4295-a344-f336c6e54c5b" containerName="payload-pre-processing" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114820 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59f4b35-8965-419a-a5e1-938ee6b4f31f" containerName="maas-api" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114826 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3edd7b-9086-42b8-a4ad-c9575db43548" containerName="payload-pre-processing" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114831 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4dd5772-0c1e-40e8-8709-d3b8ec86ddba" containerName="payload-processing" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114836 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3bbbc3-d7a2-4e54-bbc2-721a6abbcbd3" containerName="main" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114843 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="895cdd1f-c503-41e1-a4e6-383be3fb695a" containerName="payload-processing" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114849 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="912227d8-0b7e-47c0-9585-46e708095369" containerName="maas-api" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114856 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="956fe986-cddf-4077-b9df-4a4a5b93a1d3" containerName="payload-processing" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114867 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca9f387-5fb1-465d-bb4f-86a6ae431dd3" containerName="cleanup" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114878 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0775ed-89fd-4eb8-86ae-3338a74051a2" containerName="payload-pre-processing" Nov 01 04:16:48.115245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.114884 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4222c967-9ba8-446e-bb84-787570b73e28" containerName="payload-pre-processing" Nov 01 04:16:48.119446 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.119427 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.216794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.216760 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3a348f-serving-cert\"" Nov 01 04:16:48.219251 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.219232 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd"] Nov 01 04:16:48.250056 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.250007 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89975435-db64-486e-bd27-b603abc3a312-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.250243 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.250178 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-metrics-certs\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.250317 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.250276 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvg5g\" (UniqueName: \"kubernetes.io/projected/89975435-db64-486e-bd27-b603abc3a312-kube-api-access-dvg5g\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.250317 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.250308 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.374117 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.359022 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-metrics-certs\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.374542 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.374507 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvg5g\" (UniqueName: \"kubernetes.io/projected/89975435-db64-486e-bd27-b603abc3a312-kube-api-access-dvg5g\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.380750 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.380697 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.381160 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.381133 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89975435-db64-486e-bd27-b603abc3a312-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.382289 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.382223 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89975435-db64-486e-bd27-b603abc3a312-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.382601 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:48.382563 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-serving-cert: secret "maas-api-e2e-ipp-a-3a348f-serving-cert" not found Nov 01 04:16:48.382884 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:16:48.382748 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls podName:89975435-db64-486e-bd27-b603abc3a312 nodeName:}" failed. No retries permitted until 2026-11-01 04:16:48.882718796 +0000 UTC m=+2020.471813416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls") pod "maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" (UID: "89975435-db64-486e-bd27-b603abc3a312") : secret "maas-api-e2e-ipp-a-3a348f-serving-cert" not found Nov 01 04:16:48.389619 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.389213 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-metrics-certs\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.447981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.447950 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvg5g\" (UniqueName: \"kubernetes.io/projected/89975435-db64-486e-bd27-b603abc3a312-kube-api-access-dvg5g\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.538506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.538470 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912227d8-0b7e-47c0-9585-46e708095369" path="/var/lib/kubelet/pods/912227d8-0b7e-47c0-9585-46e708095369/volumes" Nov 01 04:16:48.887132 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.887097 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:48.889576 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:48.889550 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:49.032398 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:49.032349 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:49.407513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:49.407484 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd"] Nov 01 04:16:49.409343 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:16:49.409315 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89975435_db64_486e_bd27_b603abc3a312.slice/crio-44e553994984aac3adbac0780ba931574975cc16162e6e50459ac659c7bad4fa WatchSource:0}: Error finding container 44e553994984aac3adbac0780ba931574975cc16162e6e50459ac659c7bad4fa: Status 404 returned error can't find the container with id 44e553994984aac3adbac0780ba931574975cc16162e6e50459ac659c7bad4fa Nov 01 04:16:49.410808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:49.410788 6735 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 04:16:49.945417 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:49.945382 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" event={"ID":"89975435-db64-486e-bd27-b603abc3a312","Type":"ContainerStarted","Data":"44e553994984aac3adbac0780ba931574975cc16162e6e50459ac659c7bad4fa"} Nov 01 04:16:50.952544 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:50.952506 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" event={"ID":"89975435-db64-486e-bd27-b603abc3a312","Type":"ContainerStarted","Data":"89be19d9a55cf490e1f45509d2a1b83f9f94276e6d2de5ac6eee048b1b8b7e9d"} Nov 01 04:16:50.952921 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:50.952585 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:51.068384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:51.068324 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" podStartSLOduration=2.143499176 podStartE2EDuration="3.068308875s" totalImagesPullingTime="924.809699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:16:51.066587504 +0000 UTC m=+2022.655682131" watchObservedRunningTime="2026-11-01 04:16:51.068308875 +0000 UTC m=+2022.657403504" Nov 01 04:16:55.764007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:55.763965 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl"] Nov 01 04:16:55.764524 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:55.764258 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" containerID="cri-o://6a951528afff03326deed920d0c0b609b98636aaf5de7959feacfd913cef138e" gracePeriod=30 Nov 01 04:16:56.965588 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:56.965560 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:16:59.923699 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:59.923660 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv"] Nov 01 04:16:59.927746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:59.927724 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:16:59.976787 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:16:59.976754 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-131718-dockercfg-kttj4\"" Nov 01 04:17:00.023214 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.023182 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-131718\"" Nov 01 04:17:00.025822 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.025798 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv"] Nov 01 04:17:00.035738 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.035711 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b94e72f-f79b-4be1-bd80-b017b412f0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:00.035924 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.035905 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28czg\" (UniqueName: \"kubernetes.io/projected/5b94e72f-f79b-4be1-bd80-b017b412f0bc-kube-api-access-28czg\") pod \"payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:00.137840 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.137799 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b94e72f-f79b-4be1-bd80-b017b412f0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:00.138015 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.137985 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28czg\" (UniqueName: \"kubernetes.io/projected/5b94e72f-f79b-4be1-bd80-b017b412f0bc-kube-api-access-28czg\") pod \"payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:00.138658 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.138635 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b94e72f-f79b-4be1-bd80-b017b412f0bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:00.196399 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.196318 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28czg\" (UniqueName: \"kubernetes.io/projected/5b94e72f-f79b-4be1-bd80-b017b412f0bc-kube-api-access-28czg\") pod \"payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:00.241402 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.241366 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:00.611013 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.610985 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv"] Nov 01 04:17:00.612941 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:00.612913 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b94e72f_f79b_4be1_bd80_b017b412f0bc.slice/crio-aac1e50f3e37ded33372cd27b934227c58ae965446d53a7b3b2b98a53afb68b3 WatchSource:0}: Error finding container aac1e50f3e37ded33372cd27b934227c58ae965446d53a7b3b2b98a53afb68b3: Status 404 returned error can't find the container with id aac1e50f3e37ded33372cd27b934227c58ae965446d53a7b3b2b98a53afb68b3 Nov 01 04:17:00.616747 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.616725 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv"] Nov 01 04:17:00.621866 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.621848 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:00.715949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.715917 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv"] Nov 01 04:17:00.745085 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.745024 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3aec01-79aa-48a7-98be-34abc62658d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:00.745261 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.745098 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbch\" (UniqueName: \"kubernetes.io/projected/9e3aec01-79aa-48a7-98be-34abc62658d6-kube-api-access-cpbch\") pod \"payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:00.846042 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.846005 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3aec01-79aa-48a7-98be-34abc62658d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:00.846213 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.846092 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbch\" (UniqueName: \"kubernetes.io/projected/9e3aec01-79aa-48a7-98be-34abc62658d6-kube-api-access-cpbch\") pod \"payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:00.846845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.846818 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3aec01-79aa-48a7-98be-34abc62658d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:00.904244 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.904206 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbch\" (UniqueName: \"kubernetes.io/projected/9e3aec01-79aa-48a7-98be-34abc62658d6-kube-api-access-cpbch\") pod \"payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:00.961238 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:00.961204 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:01.030382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.030353 6735 generic.go:356] "Generic (PLEG): container finished" podID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerID="6a951528afff03326deed920d0c0b609b98636aaf5de7959feacfd913cef138e" exitCode=0 Nov 01 04:17:01.030576 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.030427 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" event={"ID":"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc","Type":"ContainerDied","Data":"6a951528afff03326deed920d0c0b609b98636aaf5de7959feacfd913cef138e"} Nov 01 04:17:01.030576 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.030526 6735 scope.go:122] "RemoveContainer" containerID="bbdb520cdda18265b0f29918040950a03f584822d0b8316d293b21f865b82e40" Nov 01 04:17:01.032350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.032315 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" event={"ID":"5b94e72f-f79b-4be1-bd80-b017b412f0bc","Type":"ContainerStarted","Data":"1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6"} Nov 01 04:17:01.032472 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.032367 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" event={"ID":"5b94e72f-f79b-4be1-bd80-b017b412f0bc","Type":"ContainerStarted","Data":"aac1e50f3e37ded33372cd27b934227c58ae965446d53a7b3b2b98a53afb68b3"} Nov 01 04:17:01.033213 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.033189 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:01.035127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.035102 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:17:01.144396 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.144302 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" podStartSLOduration=2.144287433 podStartE2EDuration="2.144287433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:01.142673531 +0000 UTC m=+2032.731768181" watchObservedRunningTime="2026-11-01 04:17:01.144287433 +0000 UTC m=+2032.733382090" Nov 01 04:17:01.150943 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.150919 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-certs\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151125 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.150950 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-data\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151125 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.150980 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istiod-ca-cert\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151125 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151014 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-token\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-token\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151127 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-socket\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151197 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-credential-socket\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151237 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-data" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:01.151295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151251 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-podinfo\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151260 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-certs" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:01.151295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151280 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-kube-api-access-qf4nd\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-kube-api-access-qf4nd\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151389 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-envoy\") pod \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\" (UID: \"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc\") " Nov 01 04:17:01.151622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151443 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istiod-ca-cert" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:17:01.151622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151537 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-credential-socket" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:01.151622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.151564 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-socket" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:01.152557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.152004 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.152557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.152029 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.152557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.152061 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.152557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.152075 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.152557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.152089 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.153781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.153758 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-envoy" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:01.153911 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.153821 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-kube-api-access-qf4nd" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "kube-api-access-qf4nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:17:01.154189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.154170 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-podinfo" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:17:01.154256 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.154186 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-token" pod "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" (UID: "4fed2dc4-159e-4e21-a2ac-ada2f5b943dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:17:01.291651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.270933 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.291651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.270975 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf4nd\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-kube-api-access-qf4nd\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.291651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.270992 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.291651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.271006 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:01.354103 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:01.354068 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv"] Nov 01 04:17:01.354953 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:01.354930 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3aec01_79aa_48a7_98be_34abc62658d6.slice/crio-012c860cf80fb008059bfda91310e5117cb0bd86574e90b197b19963ba5d035e WatchSource:0}: Error finding container 012c860cf80fb008059bfda91310e5117cb0bd86574e90b197b19963ba5d035e: Status 404 returned error can't find the container with id 012c860cf80fb008059bfda91310e5117cb0bd86574e90b197b19963ba5d035e Nov 01 04:17:02.040449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.040417 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" Nov 01 04:17:02.040933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.040447 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl" event={"ID":"4fed2dc4-159e-4e21-a2ac-ada2f5b943dc","Type":"ContainerDied","Data":"b4ce2995f1298cb23a77f7fc3613e084192b644406497497929b7a02f9cd019d"} Nov 01 04:17:02.040933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.040507 6735 scope.go:122] "RemoveContainer" containerID="6a951528afff03326deed920d0c0b609b98636aaf5de7959feacfd913cef138e" Nov 01 04:17:02.042249 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.042225 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" event={"ID":"9e3aec01-79aa-48a7-98be-34abc62658d6","Type":"ContainerStarted","Data":"61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff"} Nov 01 04:17:02.042362 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.042278 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" event={"ID":"9e3aec01-79aa-48a7-98be-34abc62658d6","Type":"ContainerStarted","Data":"012c860cf80fb008059bfda91310e5117cb0bd86574e90b197b19963ba5d035e"} Nov 01 04:17:02.266169 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.266112 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" podStartSLOduration=2.26609777 podStartE2EDuration="2.26609777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:02.26439407 +0000 UTC m=+2033.853488699" watchObservedRunningTime="2026-11-01 04:17:02.26609777 +0000 UTC m=+2033.855192399" Nov 01 04:17:02.388860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.387518 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl"] Nov 01 04:17:02.407417 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.405642 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5b71cd-openshift-default-77b9696d87-q8cdl"] Nov 01 04:17:02.527187 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:02.527155 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" path="/var/lib/kubelet/pods/4fed2dc4-159e-4e21-a2ac-ada2f5b943dc/volumes" Nov 01 04:17:03.000816 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:03.000777 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:17:03.050090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:03.050060 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:11.111972 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.111937 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n"] Nov 01 04:17:11.112411 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.112162 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" podUID="41574812-6157-45f7-a67b-e952c758491b" containerName="maas-api" containerID="cri-o://bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5" gracePeriod=30 Nov 01 04:17:11.417349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.417324 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:17:11.572087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.563328 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5"] Nov 01 04:17:11.572611 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.572573 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" containerID="cri-o://a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603" gracePeriod=30 Nov 01 04:17:11.610813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.610779 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41574812-6157-45f7-a67b-e952c758491b-kube-api-access-srhtl\" (UniqueName: \"kubernetes.io/projected/41574812-6157-45f7-a67b-e952c758491b-kube-api-access-srhtl\") pod \"41574812-6157-45f7-a67b-e952c758491b\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " Nov 01 04:17:11.611001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.610852 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41574812-6157-45f7-a67b-e952c758491b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41574812-6157-45f7-a67b-e952c758491b-openshift-service-ca\") pod \"41574812-6157-45f7-a67b-e952c758491b\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " Nov 01 04:17:11.611001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.610882 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls\") pod \"41574812-6157-45f7-a67b-e952c758491b\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " Nov 01 04:17:11.611001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.610987 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-metrics-certs\") pod \"41574812-6157-45f7-a67b-e952c758491b\" (UID: \"41574812-6157-45f7-a67b-e952c758491b\") " Nov 01 04:17:11.611292 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.611268 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41574812-6157-45f7-a67b-e952c758491b-openshift-service-ca" pod "41574812-6157-45f7-a67b-e952c758491b" (UID: "41574812-6157-45f7-a67b-e952c758491b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:17:11.611403 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.611388 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41574812-6157-45f7-a67b-e952c758491b-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:11.613280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.613255 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls" pod "41574812-6157-45f7-a67b-e952c758491b" (UID: "41574812-6157-45f7-a67b-e952c758491b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:17:11.613371 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.613279 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-metrics-certs" pod "41574812-6157-45f7-a67b-e952c758491b" (UID: "41574812-6157-45f7-a67b-e952c758491b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:17:11.613428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.613398 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41574812-6157-45f7-a67b-e952c758491b-kube-api-access-srhtl" pod "41574812-6157-45f7-a67b-e952c758491b" (UID: "41574812-6157-45f7-a67b-e952c758491b"). InnerVolumeSpecName "kube-api-access-srhtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:17:11.712586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.712548 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhtl\" (UniqueName: \"kubernetes.io/projected/41574812-6157-45f7-a67b-e952c758491b-kube-api-access-srhtl\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:11.712586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.712583 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:11.712789 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:11.712596 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41574812-6157-45f7-a67b-e952c758491b-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:12.134284 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.134172 6735 generic.go:356] "Generic (PLEG): container finished" podID="41574812-6157-45f7-a67b-e952c758491b" containerID="bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5" exitCode=0 Nov 01 04:17:12.134782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.134384 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" event={"ID":"41574812-6157-45f7-a67b-e952c758491b","Type":"ContainerDied","Data":"bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5"} Nov 01 04:17:12.134782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.134438 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" event={"ID":"41574812-6157-45f7-a67b-e952c758491b","Type":"ContainerDied","Data":"f430fc9064b340eca0369f538e6a7e7283c91b0378916f1c801e7b3a34dcc816"} Nov 01 04:17:12.134782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.134470 6735 scope.go:122] "RemoveContainer" containerID="bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5" Nov 01 04:17:12.134782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.134641 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n" Nov 01 04:17:12.157595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.157566 6735 scope.go:122] "RemoveContainer" containerID="bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5" Nov 01 04:17:12.158588 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:12.158561 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5\": container with ID starting with bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5 not found: ID does not exist" containerID="bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5" Nov 01 04:17:12.158695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.158596 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5"} err="failed to get container status \"bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5\": rpc error: code = NotFound desc = could not find container \"bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5\": container with ID starting with bf6e36e8a33439b7b61a458b574d43b92012879441e8565f104841e4151fe9d5 not found: ID does not exist" Nov 01 04:17:12.172429 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:12.172396 6735 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41574812_6157_45f7_a67b_e952c758491b.slice/crio-f430fc9064b340eca0369f538e6a7e7283c91b0378916f1c801e7b3a34dcc816\": RecentStats: unable to find data in memory cache]" Nov 01 04:17:12.172623 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:12.172456 6735 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41574812_6157_45f7_a67b_e952c758491b.slice/crio-f430fc9064b340eca0369f538e6a7e7283c91b0378916f1c801e7b3a34dcc816\": RecentStats: unable to find data in memory cache]" Nov 01 04:17:12.291683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.291641 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n"] Nov 01 04:17:12.295742 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.295715 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-535e84-6fcfdd9bfc-dpn5n"] Nov 01 04:17:12.526633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.526596 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41574812-6157-45f7-a67b-e952c758491b" path="/var/lib/kubelet/pods/41574812-6157-45f7-a67b-e952c758491b/volumes" Nov 01 04:17:12.695940 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.695909 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9"] Nov 01 04:17:12.696563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696537 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" Nov 01 04:17:12.696563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696560 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" Nov 01 04:17:12.696772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696596 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" Nov 01 04:17:12.696772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696605 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" Nov 01 04:17:12.696772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696654 6735 cpu_manager.go:414] "containerMap: removing container" podUID="41574812-6157-45f7-a67b-e952c758491b" containerName="maas-api" Nov 01 04:17:12.696772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696660 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41574812-6157-45f7-a67b-e952c758491b" containerName="maas-api" Nov 01 04:17:12.696772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696763 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" Nov 01 04:17:12.696772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696775 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="41574812-6157-45f7-a67b-e952c758491b" containerName="maas-api" Nov 01 04:17:12.696983 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.696784 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fed2dc4-159e-4e21-a2ac-ada2f5b943dc" containerName="istio-proxy" Nov 01 04:17:12.701400 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.701382 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.726499 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.726471 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvml8\" (UniqueName: \"kubernetes.io/projected/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-kube-api-access-nvml8\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.726652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.726596 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.726717 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.726665 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.726717 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.726703 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.753174 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.753145 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-601a4b-serving-cert\"" Nov 01 04:17:12.756170 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.756147 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9"] Nov 01 04:17:12.828061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.827963 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.828061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.827999 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.828240 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.828138 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvml8\" (UniqueName: \"kubernetes.io/projected/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-kube-api-access-nvml8\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.828240 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.828203 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.828344 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:12.828328 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-serving-cert: secret "maas-api-e2e-ipp-a-601a4b-serving-cert" not found Nov 01 04:17:12.828412 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:12.828402 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls podName:b6ab6e5f-8227-44f6-84e3-921ef42ffe3e nodeName:}" failed. No retries permitted until 2026-11-01 04:17:13.328381297 +0000 UTC m=+2044.917475922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls") pod "maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" (UID: "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e") : secret "maas-api-e2e-ipp-a-601a4b-serving-cert" not found Nov 01 04:17:12.828682 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.828667 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.831004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.830980 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-metrics-certs\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:12.888622 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:12.888597 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvml8\" (UniqueName: \"kubernetes.io/projected/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-kube-api-access-nvml8\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:13.052965 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:13.052932 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:17:13.339062 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:13.339003 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:13.341608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:13.341589 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:13.614299 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:13.614213 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:13.983526 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:13.983498 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9"] Nov 01 04:17:13.985903 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:13.985871 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ab6e5f_8227_44f6_84e3_921ef42ffe3e.slice/crio-bde2b673ca4669f391c8d6c781aa25daad5cb6170204ffb03c1e46d39278bcca WatchSource:0}: Error finding container bde2b673ca4669f391c8d6c781aa25daad5cb6170204ffb03c1e46d39278bcca: Status 404 returned error can't find the container with id bde2b673ca4669f391c8d6c781aa25daad5cb6170204ffb03c1e46d39278bcca Nov 01 04:17:14.057739 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:14.057710 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:17:14.157213 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:14.157178 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" event={"ID":"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e","Type":"ContainerStarted","Data":"bde2b673ca4669f391c8d6c781aa25daad5cb6170204ffb03c1e46d39278bcca"} Nov 01 04:17:15.164868 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:15.164832 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" event={"ID":"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e","Type":"ContainerStarted","Data":"246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43"} Nov 01 04:17:15.165364 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:15.164893 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:15.278712 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:15.278662 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" podStartSLOduration=2.333450542 podStartE2EDuration="3.278648153s" totalImagesPullingTime="945.197611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:15.276341496 +0000 UTC m=+2046.865436122" watchObservedRunningTime="2026-11-01 04:17:15.278648153 +0000 UTC m=+2046.867742781" Nov 01 04:17:15.799327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:15.799292 6735 patch_prober.go:29] interesting pod/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:17:15.799663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:15.799355 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:17:16.862099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.861686 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:17:16.897557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897521 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-socket\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897733 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897621 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-istio-token\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-istio-token\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897733 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897659 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-data\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897733 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897686 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-credential-socket\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897733 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897721 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-certs\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897753 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893283b0-538c-4124-95ee-5a36ddc405f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893283b0-538c-4124-95ee-5a36ddc405f5-istiod-ca-cert\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897814 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-envoy\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897811 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-socket" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:16.897938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897856 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-kube-api-access-bt464\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-kube-api-access-bt464\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.897938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897904 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/893283b0-538c-4124-95ee-5a36ddc405f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893283b0-538c-4124-95ee-5a36ddc405f5-istio-podinfo\") pod \"893283b0-538c-4124-95ee-5a36ddc405f5\" (UID: \"893283b0-538c-4124-95ee-5a36ddc405f5\") " Nov 01 04:17:16.898242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.897947 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-data" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:16.898242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898024 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-certs" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:16.898242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898080 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-credential-socket" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:16.898390 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898280 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893283b0-538c-4124-95ee-5a36ddc405f5-istiod-ca-cert" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:17:16.898679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898581 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:16.898679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898611 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893283b0-538c-4124-95ee-5a36ddc405f5-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:16.898679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898625 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:16.898679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898639 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:16.898679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.898652 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:16.900218 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.900188 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-envoy" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:17:16.900318 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.900257 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-istio-token" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:17:16.900364 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.900322 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/893283b0-538c-4124-95ee-5a36ddc405f5-istio-podinfo" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:17:16.900364 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.900342 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-kube-api-access-bt464" pod "893283b0-538c-4124-95ee-5a36ddc405f5" (UID: "893283b0-538c-4124-95ee-5a36ddc405f5"). InnerVolumeSpecName "kube-api-access-bt464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:17:16.999939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.999901 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt464\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-kube-api-access-bt464\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:16.999939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.999937 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893283b0-538c-4124-95ee-5a36ddc405f5-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:16.999939 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.999948 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893283b0-538c-4124-95ee-5a36ddc405f5-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:17.000217 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:16.999957 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893283b0-538c-4124-95ee-5a36ddc405f5-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:17.177977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.177882 6735 generic.go:356] "Generic (PLEG): container finished" podID="893283b0-538c-4124-95ee-5a36ddc405f5" containerID="a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603" exitCode=0 Nov 01 04:17:17.177977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.177952 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" Nov 01 04:17:17.177977 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.177965 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" event={"ID":"893283b0-538c-4124-95ee-5a36ddc405f5","Type":"ContainerDied","Data":"a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603"} Nov 01 04:17:17.178243 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.178067 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5" event={"ID":"893283b0-538c-4124-95ee-5a36ddc405f5","Type":"ContainerDied","Data":"c559a665f1ab34be0d6a342815ac1bcd23691561e3426dd2f77ebfc2857c2e99"} Nov 01 04:17:17.178243 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.178091 6735 scope.go:122] "RemoveContainer" containerID="a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603" Nov 01 04:17:17.188474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.188456 6735 scope.go:122] "RemoveContainer" containerID="a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603" Nov 01 04:17:17.188810 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:17.188790 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603\": container with ID starting with a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603 not found: ID does not exist" containerID="a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603" Nov 01 04:17:17.188862 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.188821 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603"} err="failed to get container status \"a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603\": rpc error: code = NotFound desc = could not find container \"a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603\": container with ID starting with a78f70d219d46048089c89a577fae4f1bb87b18bd2ee4a2d99fec2237a079603 not found: ID does not exist" Nov 01 04:17:17.332783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.332750 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5"] Nov 01 04:17:17.336732 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:17.336703 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-865066-openshift-default-6559876c99-jx2s5"] Nov 01 04:17:18.526975 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:18.526944 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" path="/var/lib/kubelet/pods/893283b0-538c-4124-95ee-5a36ddc405f5/volumes" Nov 01 04:17:21.177019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:21.176989 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:17:23.198029 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.197992 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz"] Nov 01 04:17:23.198521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.198215 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" podUID="4fef66ce-be48-4e3a-8770-87c5cee85404" containerName="maas-api" containerID="cri-o://0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8" gracePeriod=30 Nov 01 04:17:23.527309 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.527277 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:17:23.595902 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.595863 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fef66ce-be48-4e3a-8770-87c5cee85404-kube-api-access-p9ggv\" (UniqueName: \"kubernetes.io/projected/4fef66ce-be48-4e3a-8770-87c5cee85404-kube-api-access-p9ggv\") pod \"4fef66ce-be48-4e3a-8770-87c5cee85404\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " Nov 01 04:17:23.596147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.595955 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls\") pod \"4fef66ce-be48-4e3a-8770-87c5cee85404\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " Nov 01 04:17:23.596147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.596133 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fef66ce-be48-4e3a-8770-87c5cee85404-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fef66ce-be48-4e3a-8770-87c5cee85404-openshift-service-ca\") pod \"4fef66ce-be48-4e3a-8770-87c5cee85404\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " Nov 01 04:17:23.596269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.596173 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-metrics-certs\") pod \"4fef66ce-be48-4e3a-8770-87c5cee85404\" (UID: \"4fef66ce-be48-4e3a-8770-87c5cee85404\") " Nov 01 04:17:23.596566 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.596525 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fef66ce-be48-4e3a-8770-87c5cee85404-openshift-service-ca" pod "4fef66ce-be48-4e3a-8770-87c5cee85404" (UID: "4fef66ce-be48-4e3a-8770-87c5cee85404"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:17:23.596731 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.596704 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fef66ce-be48-4e3a-8770-87c5cee85404-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:23.598457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.598432 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-metrics-certs" pod "4fef66ce-be48-4e3a-8770-87c5cee85404" (UID: "4fef66ce-be48-4e3a-8770-87c5cee85404"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:17:23.598768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.598734 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls" pod "4fef66ce-be48-4e3a-8770-87c5cee85404" (UID: "4fef66ce-be48-4e3a-8770-87c5cee85404"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:17:23.598768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.598740 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fef66ce-be48-4e3a-8770-87c5cee85404-kube-api-access-p9ggv" pod "4fef66ce-be48-4e3a-8770-87c5cee85404" (UID: "4fef66ce-be48-4e3a-8770-87c5cee85404"). InnerVolumeSpecName "kube-api-access-p9ggv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:17:23.698335 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.698296 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ggv\" (UniqueName: \"kubernetes.io/projected/4fef66ce-be48-4e3a-8770-87c5cee85404-kube-api-access-p9ggv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:23.698335 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.698330 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:23.698335 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:23.698340 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fef66ce-be48-4e3a-8770-87c5cee85404-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:17:24.224866 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.224826 6735 generic.go:356] "Generic (PLEG): container finished" podID="4fef66ce-be48-4e3a-8770-87c5cee85404" containerID="0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8" exitCode=0 Nov 01 04:17:24.225389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.224872 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" Nov 01 04:17:24.225389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.224881 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" event={"ID":"4fef66ce-be48-4e3a-8770-87c5cee85404","Type":"ContainerDied","Data":"0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8"} Nov 01 04:17:24.225389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.224940 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz" event={"ID":"4fef66ce-be48-4e3a-8770-87c5cee85404","Type":"ContainerDied","Data":"cc9cb23181df29ecfc5690633f4b1acc5b9ca2728437f6c703aea5cfaa5fa167"} Nov 01 04:17:24.225389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.224970 6735 scope.go:122] "RemoveContainer" containerID="0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8" Nov 01 04:17:24.235169 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.235149 6735 scope.go:122] "RemoveContainer" containerID="0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8" Nov 01 04:17:24.235429 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:24.235409 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8\": container with ID starting with 0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8 not found: ID does not exist" containerID="0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8" Nov 01 04:17:24.235495 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.235437 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8"} err="failed to get container status \"0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8\": rpc error: code = NotFound desc = could not find container \"0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8\": container with ID starting with 0a912e590958ae6700c5c0fdf63375b881df166cf77ae290fbfbeb0e813e45c8 not found: ID does not exist" Nov 01 04:17:24.384447 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.384409 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz"] Nov 01 04:17:24.430312 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.430276 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d683e-685bc76475-4qbgz"] Nov 01 04:17:24.525360 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:24.525282 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fef66ce-be48-4e3a-8770-87c5cee85404" path="/var/lib/kubelet/pods/4fef66ce-be48-4e3a-8770-87c5cee85404/volumes" Nov 01 04:17:25.943092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.943027 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b"] Nov 01 04:17:25.943618 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.943604 6735 cpu_manager.go:414] "containerMap: removing container" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" Nov 01 04:17:25.943678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.943619 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" Nov 01 04:17:25.943678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.943646 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4fef66ce-be48-4e3a-8770-87c5cee85404" containerName="maas-api" Nov 01 04:17:25.943678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.943651 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fef66ce-be48-4e3a-8770-87c5cee85404" containerName="maas-api" Nov 01 04:17:25.943784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.943745 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fef66ce-be48-4e3a-8770-87c5cee85404" containerName="maas-api" Nov 01 04:17:25.943784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.943764 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="893283b0-538c-4124-95ee-5a36ddc405f5" containerName="istio-proxy" Nov 01 04:17:25.948436 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:25.948420 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.048334 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.048292 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a7b5f0\"" Nov 01 04:17:26.055006 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.054972 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a7b5f0-dockercfg-pftxp\"" Nov 01 04:17:26.056091 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.055970 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fndv\" (UniqueName: \"kubernetes.io/projected/9d223186-7714-4b83-b5f1-d393853da167-kube-api-access-2fndv\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.056329 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.056299 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d223186-7714-4b83-b5f1-d393853da167-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.063270 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.063238 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b"] Nov 01 04:17:26.104471 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.099911 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls"] Nov 01 04:17:26.132229 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.132197 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.158026 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.157997 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d223186-7714-4b83-b5f1-d393853da167-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.158216 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.158178 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fndv\" (UniqueName: \"kubernetes.io/projected/9d223186-7714-4b83-b5f1-d393853da167-kube-api-access-2fndv\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.158749 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.158730 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d223186-7714-4b83-b5f1-d393853da167-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.168893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.168862 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls"] Nov 01 04:17:26.216127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.216100 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fndv\" (UniqueName: \"kubernetes.io/projected/9d223186-7714-4b83-b5f1-d393853da167-kube-api-access-2fndv\") pod \"payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.259640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.259604 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjdt\" (UniqueName: \"kubernetes.io/projected/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-kube-api-access-gmjdt\") pod \"payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.259831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.259800 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.262309 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.262285 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:26.361671 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.361635 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjdt\" (UniqueName: \"kubernetes.io/projected/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-kube-api-access-gmjdt\") pod \"payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.361871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.361754 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.362711 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.362653 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.419141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.419111 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjdt\" (UniqueName: \"kubernetes.io/projected/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-kube-api-access-gmjdt\") pod \"payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.447966 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.447937 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:26.635160 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.635132 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b"] Nov 01 04:17:26.637163 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:26.637137 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d223186_7714_4b83_b5f1_d393853da167.slice/crio-06479e8df0b5e93c4aaf295c8684d04565df68ce8f14165d55f1363e1b947d08 WatchSource:0}: Error finding container 06479e8df0b5e93c4aaf295c8684d04565df68ce8f14165d55f1363e1b947d08: Status 404 returned error can't find the container with id 06479e8df0b5e93c4aaf295c8684d04565df68ce8f14165d55f1363e1b947d08 Nov 01 04:17:26.816016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:26.815993 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls"] Nov 01 04:17:26.818109 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:26.818081 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb622d8c4_d5ce_4ec2_be3f_c8fd4a0fb9d6.slice/crio-5f8e78e7e7d8659e1ce5b2c36629d0c13c4119a960cb632abf990270eb71087f WatchSource:0}: Error finding container 5f8e78e7e7d8659e1ce5b2c36629d0c13c4119a960cb632abf990270eb71087f: Status 404 returned error can't find the container with id 5f8e78e7e7d8659e1ce5b2c36629d0c13c4119a960cb632abf990270eb71087f Nov 01 04:17:27.246404 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.246365 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" event={"ID":"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6","Type":"ContainerStarted","Data":"3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2"} Nov 01 04:17:27.246867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.246428 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" event={"ID":"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6","Type":"ContainerStarted","Data":"5f8e78e7e7d8659e1ce5b2c36629d0c13c4119a960cb632abf990270eb71087f"} Nov 01 04:17:27.246867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.246588 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:27.247893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.247869 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" event={"ID":"9d223186-7714-4b83-b5f1-d393853da167","Type":"ContainerStarted","Data":"53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755"} Nov 01 04:17:27.248024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.247910 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" event={"ID":"9d223186-7714-4b83-b5f1-d393853da167","Type":"ContainerStarted","Data":"06479e8df0b5e93c4aaf295c8684d04565df68ce8f14165d55f1363e1b947d08"} Nov 01 04:17:27.248024 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.247937 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:27.357544 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.357494 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" podStartSLOduration=1.357480741 podStartE2EDuration="1.357480741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:27.355167413 +0000 UTC m=+2058.944262041" watchObservedRunningTime="2026-11-01 04:17:27.357480741 +0000 UTC m=+2058.946575370" Nov 01 04:17:27.495307 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:27.495237 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" podStartSLOduration=2.495216959 podStartE2EDuration="2.495216959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:27.489818988 +0000 UTC m=+2059.078913618" watchObservedRunningTime="2026-11-01 04:17:27.495216959 +0000 UTC m=+2059.084311589" Nov 01 04:17:28.495082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:28.491196 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:17:31.758771 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.758728 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx"] Nov 01 04:17:31.763967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.763942 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:31.815358 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.815310 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4c2a66\"" Nov 01 04:17:31.850587 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.850555 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e528ab-13d2-4e11-bd51-bb2141f465bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:31.850777 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.850663 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvcl\" (UniqueName: \"kubernetes.io/projected/b4e528ab-13d2-4e11-bd51-bb2141f465bf-kube-api-access-8kvcl\") pod \"payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:31.864529 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.864501 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx"] Nov 01 04:17:31.937587 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.937548 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd"] Nov 01 04:17:31.942994 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.942966 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:31.965063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.963988 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rnd7\" (UniqueName: \"kubernetes.io/projected/c9c74d9c-c29c-4943-adaf-34dc271e5afc-kube-api-access-4rnd7\") pod \"payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:31.965063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.964194 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e528ab-13d2-4e11-bd51-bb2141f465bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:31.965063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.964267 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74d9c-c29c-4943-adaf-34dc271e5afc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:31.965063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.964335 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvcl\" (UniqueName: \"kubernetes.io/projected/b4e528ab-13d2-4e11-bd51-bb2141f465bf-kube-api-access-8kvcl\") pod \"payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:31.965063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.965015 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e528ab-13d2-4e11-bd51-bb2141f465bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:31.987155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.987117 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd"] Nov 01 04:17:31.992426 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:31.992401 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4c2a66-dockercfg-5lfml\"" Nov 01 04:17:32.024284 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.024195 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvcl\" (UniqueName: \"kubernetes.io/projected/b4e528ab-13d2-4e11-bd51-bb2141f465bf-kube-api-access-8kvcl\") pod \"payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:32.065772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.065737 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnd7\" (UniqueName: \"kubernetes.io/projected/c9c74d9c-c29c-4943-adaf-34dc271e5afc-kube-api-access-4rnd7\") pod \"payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:32.065951 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.065792 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74d9c-c29c-4943-adaf-34dc271e5afc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:32.066484 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.066462 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74d9c-c29c-4943-adaf-34dc271e5afc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:32.079153 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.079128 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:32.124506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.124468 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnd7\" (UniqueName: \"kubernetes.io/projected/c9c74d9c-c29c-4943-adaf-34dc271e5afc-kube-api-access-4rnd7\") pod \"payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:32.279384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.279298 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:32.453131 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.453100 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx"] Nov 01 04:17:32.454868 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:32.454842 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e528ab_13d2_4e11_bd51_bb2141f465bf.slice/crio-75ebf5dcf7e4a01b125bc65daa8ec825cf3016f375f6704141608f0fd00b92c5 WatchSource:0}: Error finding container 75ebf5dcf7e4a01b125bc65daa8ec825cf3016f375f6704141608f0fd00b92c5: Status 404 returned error can't find the container with id 75ebf5dcf7e4a01b125bc65daa8ec825cf3016f375f6704141608f0fd00b92c5 Nov 01 04:17:32.652433 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:32.652400 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd"] Nov 01 04:17:32.654197 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:32.654168 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c74d9c_c29c_4943_adaf_34dc271e5afc.slice/crio-d8cc98f764a858bc6a1d13d2012bfad4ae96e8e580d76c5b3ffd3c2fb69b631b WatchSource:0}: Error finding container d8cc98f764a858bc6a1d13d2012bfad4ae96e8e580d76c5b3ffd3c2fb69b631b: Status 404 returned error can't find the container with id d8cc98f764a858bc6a1d13d2012bfad4ae96e8e580d76c5b3ffd3c2fb69b631b Nov 01 04:17:33.001841 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:17:33.001795 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:17:33.308429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.308113 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" event={"ID":"c9c74d9c-c29c-4943-adaf-34dc271e5afc","Type":"ContainerStarted","Data":"86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8"} Nov 01 04:17:33.308429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.308260 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" event={"ID":"c9c74d9c-c29c-4943-adaf-34dc271e5afc","Type":"ContainerStarted","Data":"d8cc98f764a858bc6a1d13d2012bfad4ae96e8e580d76c5b3ffd3c2fb69b631b"} Nov 01 04:17:33.308429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.308342 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:33.310397 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.310368 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" event={"ID":"b4e528ab-13d2-4e11-bd51-bb2141f465bf","Type":"ContainerStarted","Data":"a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a"} Nov 01 04:17:33.310557 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.310423 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" event={"ID":"b4e528ab-13d2-4e11-bd51-bb2141f465bf","Type":"ContainerStarted","Data":"75ebf5dcf7e4a01b125bc65daa8ec825cf3016f375f6704141608f0fd00b92c5"} Nov 01 04:17:33.311089 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.311070 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:33.419582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.419515 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" podStartSLOduration=2.419495383 podStartE2EDuration="2.419495383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:33.417292291 +0000 UTC m=+2065.006386920" watchObservedRunningTime="2026-11-01 04:17:33.419495383 +0000 UTC m=+2065.008590013" Nov 01 04:17:33.541779 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:33.541406 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" podStartSLOduration=2.541387643 podStartE2EDuration="2.541387643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:33.536896841 +0000 UTC m=+2065.125991471" watchObservedRunningTime="2026-11-01 04:17:33.541387643 +0000 UTC m=+2065.130482276" Nov 01 04:17:35.729860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.729825 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96"] Nov 01 04:17:35.734327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.734306 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.802061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.798794 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e7c9be-openshift-default-dockercfg-kvdr9\"" Nov 01 04:17:35.850425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.850380 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96"] Nov 01 04:17:35.950957 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.950893 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-certs\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.959783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.959748 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istiod-ca-cert\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.964081 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.960114 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-287mc\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-kube-api-access-287mc\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.964746 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.964718 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-podinfo\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.964955 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.964933 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-envoy\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.965105 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.964970 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-data\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.965282 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.965259 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-token\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.965371 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.965300 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-socket\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:35.965438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:35.965414 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-credential-socket\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.066882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.066789 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istiod-ca-cert\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.066882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.066827 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-287mc\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-kube-api-access-287mc\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.066882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.066855 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-podinfo\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.066882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.066871 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-envoy\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.066889 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-data\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067020 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-token\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067083 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-socket\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067170 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-credential-socket\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067271 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067243 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-certs\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067530 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067443 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-socket\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067544 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-credential-socket\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067639 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067605 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-certs\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067699 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067667 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-data\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.067741 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.067693 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istiod-ca-cert\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.069932 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.069905 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-envoy\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.070274 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.070253 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-podinfo\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.133256 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.133224 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-token\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.133444 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.133424 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-287mc\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-kube-api-access-287mc\") pod \"e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.348021 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.347942 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:36.771235 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.771211 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96"] Nov 01 04:17:36.773243 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:36.773211 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adf4f20_5ee4_4dad_9b7f_e68f4007c068.slice/crio-a738ffc37d23d35bd0ef0e47e87a392c1a3dc66d304d76aa6e0541a83958d730 WatchSource:0}: Error finding container a738ffc37d23d35bd0ef0e47e87a392c1a3dc66d304d76aa6e0541a83958d730: Status 404 returned error can't find the container with id a738ffc37d23d35bd0ef0e47e87a392c1a3dc66d304d76aa6e0541a83958d730 Nov 01 04:17:36.775870 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.775834 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:17:36.775960 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.775906 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:17:36.775960 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:36.775934 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:17:37.335722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:37.335684 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" event={"ID":"1adf4f20-5ee4-4dad-9b7f-e68f4007c068","Type":"ContainerStarted","Data":"7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778"} Nov 01 04:17:37.335908 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:37.335767 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" event={"ID":"1adf4f20-5ee4-4dad-9b7f-e68f4007c068","Type":"ContainerStarted","Data":"a738ffc37d23d35bd0ef0e47e87a392c1a3dc66d304d76aa6e0541a83958d730"} Nov 01 04:17:37.348179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:37.348151 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:37.353866 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:37.353837 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:37.498598 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:37.498548 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podStartSLOduration=2.498535052 podStartE2EDuration="2.498535052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:37.495484103 +0000 UTC m=+2069.084578733" watchObservedRunningTime="2026-11-01 04:17:37.498535052 +0000 UTC m=+2069.087629681" Nov 01 04:17:38.262243 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:38.262219 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:17:38.262634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:38.262281 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:17:38.341827 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:38.341793 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:38.342721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:38.342704 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:17:43.871408 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:43.871380 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8"] Nov 01 04:17:43.875794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:43.875770 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:43.926560 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:43.926533 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e7c9be\"" Nov 01 04:17:43.968903 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:43.968861 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b2f327-5382-426f-a613-1b9a8d3e4c85-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:43.969134 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:43.968996 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lckw\" (UniqueName: \"kubernetes.io/projected/c9b2f327-5382-426f-a613-1b9a8d3e4c85-kube-api-access-6lckw\") pod \"payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:43.973015 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:43.972989 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e7c9be-dockercfg-qhsmm\"" Nov 01 04:17:43.976322 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:43.976290 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8"] Nov 01 04:17:44.057066 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.055704 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd"] Nov 01 04:17:44.062865 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.062829 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.070625 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.070601 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b2f327-5382-426f-a613-1b9a8d3e4c85-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:44.070971 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.070935 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lckw\" (UniqueName: \"kubernetes.io/projected/c9b2f327-5382-426f-a613-1b9a8d3e4c85-kube-api-access-6lckw\") pod \"payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:44.071556 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.071528 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b2f327-5382-426f-a613-1b9a8d3e4c85-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:44.105238 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.105201 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd"] Nov 01 04:17:44.129248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.129161 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lckw\" (UniqueName: \"kubernetes.io/projected/c9b2f327-5382-426f-a613-1b9a8d3e4c85-kube-api-access-6lckw\") pod \"payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:44.172115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.172071 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzxk\" (UniqueName: \"kubernetes.io/projected/5f561522-b6dc-43b3-bf9e-75112df1c820-kube-api-access-gbzxk\") pod \"payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.172394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.172374 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f561522-b6dc-43b3-bf9e-75112df1c820-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.190721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.190685 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:44.273938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.273904 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f561522-b6dc-43b3-bf9e-75112df1c820-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.274138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.273951 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzxk\" (UniqueName: \"kubernetes.io/projected/5f561522-b6dc-43b3-bf9e-75112df1c820-kube-api-access-gbzxk\") pod \"payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.274644 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.274624 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f561522-b6dc-43b3-bf9e-75112df1c820-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.318766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.318738 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:17:44.334996 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.334969 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzxk\" (UniqueName: \"kubernetes.io/projected/5f561522-b6dc-43b3-bf9e-75112df1c820-kube-api-access-gbzxk\") pod \"payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.379027 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.378998 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:44.565617 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.565591 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8"] Nov 01 04:17:44.567311 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:44.567279 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b2f327_5382_426f_a613_1b9a8d3e4c85.slice/crio-a1e3d59f7d4c9403dc7f0453f36653a8cb254d6c073bf7c5c80880a046989d34 WatchSource:0}: Error finding container a1e3d59f7d4c9403dc7f0453f36653a8cb254d6c073bf7c5c80880a046989d34: Status 404 returned error can't find the container with id a1e3d59f7d4c9403dc7f0453f36653a8cb254d6c073bf7c5c80880a046989d34 Nov 01 04:17:44.749092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:44.749054 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd"] Nov 01 04:17:44.751379 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:17:44.751343 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f561522_b6dc_43b3_bf9e_75112df1c820.slice/crio-27892ac41273fae0f7fdabac6eab36f0d592dfaac5a57b1fa87ee1ea1f4747f4 WatchSource:0}: Error finding container 27892ac41273fae0f7fdabac6eab36f0d592dfaac5a57b1fa87ee1ea1f4747f4: Status 404 returned error can't find the container with id 27892ac41273fae0f7fdabac6eab36f0d592dfaac5a57b1fa87ee1ea1f4747f4 Nov 01 04:17:45.324617 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.324586 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:17:45.387429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.387397 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" event={"ID":"c9b2f327-5382-426f-a613-1b9a8d3e4c85","Type":"ContainerStarted","Data":"6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9"} Nov 01 04:17:45.387623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.387457 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" event={"ID":"c9b2f327-5382-426f-a613-1b9a8d3e4c85","Type":"ContainerStarted","Data":"a1e3d59f7d4c9403dc7f0453f36653a8cb254d6c073bf7c5c80880a046989d34"} Nov 01 04:17:45.387623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.387489 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:17:45.389015 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.388989 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" event={"ID":"5f561522-b6dc-43b3-bf9e-75112df1c820","Type":"ContainerStarted","Data":"6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6"} Nov 01 04:17:45.389158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.389030 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" event={"ID":"5f561522-b6dc-43b3-bf9e-75112df1c820","Type":"ContainerStarted","Data":"27892ac41273fae0f7fdabac6eab36f0d592dfaac5a57b1fa87ee1ea1f4747f4"} Nov 01 04:17:45.389158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.389110 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:45.553810 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.552154 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" podStartSLOduration=1.552135582 podStartE2EDuration="1.552135582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:45.549238959 +0000 UTC m=+2077.138333590" watchObservedRunningTime="2026-11-01 04:17:45.552135582 +0000 UTC m=+2077.141230214" Nov 01 04:17:45.680466 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:45.680345 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" podStartSLOduration=2.680323805 podStartE2EDuration="2.680323805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:45.671481894 +0000 UTC m=+2077.260576520" watchObservedRunningTime="2026-11-01 04:17:45.680323805 +0000 UTC m=+2077.269418435" Nov 01 04:17:55.952137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:55.952102 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:17:56.397889 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:56.397859 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:17:56.398084 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:17:56.397933 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:18:01.084687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:01.084640 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7"] Nov 01 04:18:03.002877 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:18:03.002843 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:18:06.085498 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.085462 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" podUID="2ce26ea6-8551-4917-8742-161d9c8b49d9" containerName="payload-processing" containerID="cri-o://fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39" gracePeriod=25 Nov 01 04:18:06.347337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.347269 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:18:06.495712 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.495675 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce26ea6-8551-4917-8742-161d9c8b49d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce26ea6-8551-4917-8742-161d9c8b49d9-plugins-config-volume\") pod \"2ce26ea6-8551-4917-8742-161d9c8b49d9\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " Nov 01 04:18:06.495902 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.495762 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce26ea6-8551-4917-8742-161d9c8b49d9-kube-api-access-lp6f6\" (UniqueName: \"kubernetes.io/projected/2ce26ea6-8551-4917-8742-161d9c8b49d9-kube-api-access-lp6f6\") pod \"2ce26ea6-8551-4917-8742-161d9c8b49d9\" (UID: \"2ce26ea6-8551-4917-8742-161d9c8b49d9\") " Nov 01 04:18:06.496114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.496089 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce26ea6-8551-4917-8742-161d9c8b49d9-plugins-config-volume" pod "2ce26ea6-8551-4917-8742-161d9c8b49d9" (UID: "2ce26ea6-8551-4917-8742-161d9c8b49d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:18:06.498226 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.498196 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce26ea6-8551-4917-8742-161d9c8b49d9-kube-api-access-lp6f6" pod "2ce26ea6-8551-4917-8742-161d9c8b49d9" (UID: "2ce26ea6-8551-4917-8742-161d9c8b49d9"). InnerVolumeSpecName "kube-api-access-lp6f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:18:06.566599 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.566570 6735 generic.go:356] "Generic (PLEG): container finished" podID="2ce26ea6-8551-4917-8742-161d9c8b49d9" containerID="fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39" exitCode=0 Nov 01 04:18:06.566768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.566613 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" Nov 01 04:18:06.566768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.566638 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" event={"ID":"2ce26ea6-8551-4917-8742-161d9c8b49d9","Type":"ContainerDied","Data":"fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39"} Nov 01 04:18:06.566768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.566691 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7" event={"ID":"2ce26ea6-8551-4917-8742-161d9c8b49d9","Type":"ContainerDied","Data":"a6d6b85fb02f81709c74aeca277498a2ab1db1e79cdbe94ad20f80d519dab740"} Nov 01 04:18:06.566768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.566720 6735 scope.go:122] "RemoveContainer" containerID="fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39" Nov 01 04:18:06.576384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.576367 6735 scope.go:122] "RemoveContainer" containerID="fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39" Nov 01 04:18:06.576640 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:18:06.576624 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39\": container with ID starting with fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39 not found: ID does not exist" containerID="fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39" Nov 01 04:18:06.576712 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.576650 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39"} err="failed to get container status \"fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39\": rpc error: code = NotFound desc = could not find container \"fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39\": container with ID starting with fcdfb17342d02615646827d11e303ade5ea0188059dadab45a2347bffbabdf39 not found: ID does not exist" Nov 01 04:18:06.597667 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.597587 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce26ea6-8551-4917-8742-161d9c8b49d9-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:18:06.597667 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.597614 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp6f6\" (UniqueName: \"kubernetes.io/projected/2ce26ea6-8551-4917-8742-161d9c8b49d9-kube-api-access-lp6f6\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:18:06.720609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.720573 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7"] Nov 01 04:18:06.729806 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:06.729770 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-6f548bcf5-qc8b7"] Nov 01 04:18:08.527163 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:08.527128 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce26ea6-8551-4917-8742-161d9c8b49d9" path="/var/lib/kubelet/pods/2ce26ea6-8551-4917-8742-161d9c8b49d9/volumes" Nov 01 04:18:09.785614 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:09.785584 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:18:09.791671 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:09.791653 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:18:18.191491 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.191456 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq"] Nov 01 04:18:18.192097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.192070 6735 cpu_manager.go:414] "containerMap: removing container" podUID="2ce26ea6-8551-4917-8742-161d9c8b49d9" containerName="payload-processing" Nov 01 04:18:18.192097 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.192086 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce26ea6-8551-4917-8742-161d9c8b49d9" containerName="payload-processing" Nov 01 04:18:18.192220 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.192191 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce26ea6-8551-4917-8742-161d9c8b49d9" containerName="payload-processing" Nov 01 04:18:18.197208 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.197191 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.294858 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.294820 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a7b5f0-serving-cert\"" Nov 01 04:18:18.297345 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.297323 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq"] Nov 01 04:18:18.326640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.326606 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9509d859-1099-41f5-8466-aad7e8d13f8d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.326831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.326763 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmbr\" (UniqueName: \"kubernetes.io/projected/9509d859-1099-41f5-8466-aad7e8d13f8d-kube-api-access-8zmbr\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.326831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.326822 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.327017 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.326993 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-metrics-certs\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.427766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.427737 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9509d859-1099-41f5-8466-aad7e8d13f8d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.427962 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.427824 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmbr\" (UniqueName: \"kubernetes.io/projected/9509d859-1099-41f5-8466-aad7e8d13f8d-kube-api-access-8zmbr\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.427962 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.427869 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.427962 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.427937 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-metrics-certs\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.428146 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:18:18.428021 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-serving-cert: secret "maas-api-e2e-ipp-b-a7b5f0-serving-cert" not found Nov 01 04:18:18.428146 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:18:18.428099 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls podName:9509d859-1099-41f5-8466-aad7e8d13f8d nodeName:}" failed. No retries permitted until 2026-11-01 04:18:18.928077954 +0000 UTC m=+2110.517172575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls") pod "maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" (UID: "9509d859-1099-41f5-8466-aad7e8d13f8d") : secret "maas-api-e2e-ipp-b-a7b5f0-serving-cert" not found Nov 01 04:18:18.428423 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.428404 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9509d859-1099-41f5-8466-aad7e8d13f8d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.431847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.431821 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-metrics-certs\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.485972 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.485946 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmbr\" (UniqueName: \"kubernetes.io/projected/9509d859-1099-41f5-8466-aad7e8d13f8d-kube-api-access-8zmbr\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.936856 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.936772 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:18.939396 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:18.939369 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:19.111207 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:19.111165 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:19.481144 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:19.481117 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq"] Nov 01 04:18:19.482667 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:18:19.482643 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9509d859_1099_41f5_8466_aad7e8d13f8d.slice/crio-818d057890ae96aa3ab1dcf668c5ff4d196d98cb765bdb58115bd8970dde6bd7 WatchSource:0}: Error finding container 818d057890ae96aa3ab1dcf668c5ff4d196d98cb765bdb58115bd8970dde6bd7: Status 404 returned error can't find the container with id 818d057890ae96aa3ab1dcf668c5ff4d196d98cb765bdb58115bd8970dde6bd7 Nov 01 04:18:19.665445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:19.665403 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" event={"ID":"9509d859-1099-41f5-8466-aad7e8d13f8d","Type":"ContainerStarted","Data":"818d057890ae96aa3ab1dcf668c5ff4d196d98cb765bdb58115bd8970dde6bd7"} Nov 01 04:18:20.673361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:20.673277 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" event={"ID":"9509d859-1099-41f5-8466-aad7e8d13f8d","Type":"ContainerStarted","Data":"300d3a1790e66ae6a778c0efa6fa5835deb249003a127dad7d7aec58b986d371"} Nov 01 04:18:20.673730 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:20.673418 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:20.785242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:20.785191 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" podStartSLOduration=1.877540789 podStartE2EDuration="2.7851788s" totalImagesPullingTime="907.638011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:18:20.783609989 +0000 UTC m=+2112.372704618" watchObservedRunningTime="2026-11-01 04:18:20.7851788 +0000 UTC m=+2112.374273428" Nov 01 04:18:26.690208 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:26.690176 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:18:33.002504 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:18:33.002469 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:18:43.034715 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:18:43.034671 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:19:03.001309 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:19:03.001272 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:19:10.588813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.588774 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m"] Nov 01 04:19:10.593239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.593210 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:10.648490 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.648457 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m"] Nov 01 04:19:10.658629 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.658605 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc0e05-a8f1-414c-b12a-deed8686bf07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:10.658862 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.658839 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76v2p\" (UniqueName: \"kubernetes.io/projected/dacc0e05-a8f1-414c-b12a-deed8686bf07-kube-api-access-76v2p\") pod \"payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:10.691333 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.691299 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3a348f\"" Nov 01 04:19:10.691481 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.691408 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3a348f-dockercfg-bfntg\"" Nov 01 04:19:10.759547 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.759504 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76v2p\" (UniqueName: \"kubernetes.io/projected/dacc0e05-a8f1-414c-b12a-deed8686bf07-kube-api-access-76v2p\") pod \"payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:10.759745 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.759588 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc0e05-a8f1-414c-b12a-deed8686bf07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:10.760352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.760324 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc0e05-a8f1-414c-b12a-deed8686bf07-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:10.816938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.816910 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76v2p\" (UniqueName: \"kubernetes.io/projected/dacc0e05-a8f1-414c-b12a-deed8686bf07-kube-api-access-76v2p\") pod \"payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:10.924114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:10.912306 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:11.203711 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.203590 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:19:11.335902 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.335877 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m"] Nov 01 04:19:11.337226 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:19:11.337196 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacc0e05_a8f1_414c_b12a_deed8686bf07.slice/crio-5d63e7c133aed4da202cd30076732d0fd6c4eda46cce6f049110fa90a8fd75a1 WatchSource:0}: Error finding container 5d63e7c133aed4da202cd30076732d0fd6c4eda46cce6f049110fa90a8fd75a1: Status 404 returned error can't find the container with id 5d63e7c133aed4da202cd30076732d0fd6c4eda46cce6f049110fa90a8fd75a1 Nov 01 04:19:11.368457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.368432 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67"] Nov 01 04:19:11.373741 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.373723 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:11.441667 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.441628 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67"] Nov 01 04:19:11.470177 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.470145 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b6f1e-f328-405c-86b7-00f2152e5567-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:11.470393 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.470371 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4hs\" (UniqueName: \"kubernetes.io/projected/426b6f1e-f328-405c-86b7-00f2152e5567-kube-api-access-jc4hs\") pod \"payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:11.571406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.571374 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4hs\" (UniqueName: \"kubernetes.io/projected/426b6f1e-f328-405c-86b7-00f2152e5567-kube-api-access-jc4hs\") pod \"payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:11.571598 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.571461 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b6f1e-f328-405c-86b7-00f2152e5567-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:11.572212 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.572186 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b6f1e-f328-405c-86b7-00f2152e5567-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:11.629535 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.629504 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4hs\" (UniqueName: \"kubernetes.io/projected/426b6f1e-f328-405c-86b7-00f2152e5567-kube-api-access-jc4hs\") pod \"payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:11.694928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:11.694898 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:12.030179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:12.030088 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" event={"ID":"dacc0e05-a8f1-414c-b12a-deed8686bf07","Type":"ContainerStarted","Data":"4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce"} Nov 01 04:19:12.030179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:12.030154 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" event={"ID":"dacc0e05-a8f1-414c-b12a-deed8686bf07","Type":"ContainerStarted","Data":"5d63e7c133aed4da202cd30076732d0fd6c4eda46cce6f049110fa90a8fd75a1"} Nov 01 04:19:12.030179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:12.030178 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:12.064477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:12.064450 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67"] Nov 01 04:19:12.066441 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:19:12.066404 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426b6f1e_f328_405c_86b7_00f2152e5567.slice/crio-acb9994b690ade0839e35cb8c8789f17fd16418e9648eeae5d62734a23b3fdef WatchSource:0}: Error finding container acb9994b690ade0839e35cb8c8789f17fd16418e9648eeae5d62734a23b3fdef: Status 404 returned error can't find the container with id acb9994b690ade0839e35cb8c8789f17fd16418e9648eeae5d62734a23b3fdef Nov 01 04:19:12.140757 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:12.140696 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" podStartSLOduration=2.14067897 podStartE2EDuration="2.14067897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:19:12.138646941 +0000 UTC m=+2163.727741570" watchObservedRunningTime="2026-11-01 04:19:12.14067897 +0000 UTC m=+2163.729773600" Nov 01 04:19:13.038743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:13.038708 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" event={"ID":"426b6f1e-f328-405c-86b7-00f2152e5567","Type":"ContainerStarted","Data":"c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc"} Nov 01 04:19:13.039200 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:13.038769 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" event={"ID":"426b6f1e-f328-405c-86b7-00f2152e5567","Type":"ContainerStarted","Data":"acb9994b690ade0839e35cb8c8789f17fd16418e9648eeae5d62734a23b3fdef"} Nov 01 04:19:13.039200 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:13.039024 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:13.154189 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:13.154126 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" podStartSLOduration=2.15410732 podStartE2EDuration="2.15410732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:19:13.152426088 +0000 UTC m=+2164.741520717" watchObservedRunningTime="2026-11-01 04:19:13.15410732 +0000 UTC m=+2164.743201952" Nov 01 04:19:22.153757 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:22.153722 6735 generic.go:356] "Generic (PLEG): container finished" podID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerID="7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778" exitCode=0 Nov 01 04:19:22.154337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:22.153765 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" event={"ID":"1adf4f20-5ee4-4dad-9b7f-e68f4007c068","Type":"ContainerDied","Data":"7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778"} Nov 01 04:19:22.154337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:22.154243 6735 scope.go:122] "RemoveContainer" containerID="7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778" Nov 01 04:19:22.155703 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:22.155671 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:19:22.155792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:22.155761 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:19:22.155830 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:22.155794 6735 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 01 04:19:23.041435 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:23.041405 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:19:23.162558 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:23.162519 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" event={"ID":"1adf4f20-5ee4-4dad-9b7f-e68f4007c068","Type":"ContainerStarted","Data":"8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c"} Nov 01 04:19:23.342431 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:23.342340 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:19:23.349384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:23.349357 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:19:23.351067 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:23.351018 6735 patch_prober.go:29] interesting pod/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 04:19:23.351605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:23.351568 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 04:19:24.070314 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:24.070269 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:19:24.368564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:24.368463 6735 patch_prober.go:29] interesting pod/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 04:19:24.369200 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:24.369156 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 04:19:25.370382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:25.370345 6735 patch_prober.go:29] interesting pod/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 04:19:25.370799 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:25.370412 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 04:19:26.365895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:26.365403 6735 patch_prober.go:29] interesting pod/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 04:19:26.365895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:26.365506 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 04:19:27.369061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:27.368237 6735 patch_prober.go:29] interesting pod/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 04:19:27.369061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:27.368307 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 04:19:28.366060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:28.359007 6735 patch_prober.go:29] interesting pod/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 04:19:28.366060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:28.359100 6735 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 04:19:29.354017 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:29.353988 6735 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:19:29.354994 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:29.354972 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:19:33.002446 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:19:33.002412 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:19:34.470274 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.470239 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh"] Nov 01 04:19:34.474387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.474369 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:34.508371 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.508342 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52635e3-4499-422b-a147-e66dbcc499f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:34.508504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.508474 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwpt\" (UniqueName: \"kubernetes.io/projected/d52635e3-4499-422b-a147-e66dbcc499f9-kube-api-access-6kwpt\") pod \"payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:34.525879 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.525776 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-601a4b\"" Nov 01 04:19:34.572318 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.572281 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-601a4b-dockercfg-hzkvz\"" Nov 01 04:19:34.574829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.574810 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh"] Nov 01 04:19:34.609654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.609624 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52635e3-4499-422b-a147-e66dbcc499f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:34.609802 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.609716 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwpt\" (UniqueName: \"kubernetes.io/projected/d52635e3-4499-422b-a147-e66dbcc499f9-kube-api-access-6kwpt\") pod \"payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:34.610436 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.610419 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52635e3-4499-422b-a147-e66dbcc499f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:34.667873 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.667840 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwpt\" (UniqueName: \"kubernetes.io/projected/d52635e3-4499-422b-a147-e66dbcc499f9-kube-api-access-6kwpt\") pod \"payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:34.788211 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:34.788138 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:35.157803 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:35.157772 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh"] Nov 01 04:19:35.159721 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:19:35.159695 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52635e3_4499_422b_a147_e66dbcc499f9.slice/crio-9d5e328712b0f4245f39ec1a9043a9b51ac6ab003c9f2b92a3a1adb61f2b26da WatchSource:0}: Error finding container 9d5e328712b0f4245f39ec1a9043a9b51ac6ab003c9f2b92a3a1adb61f2b26da: Status 404 returned error can't find the container with id 9d5e328712b0f4245f39ec1a9043a9b51ac6ab003c9f2b92a3a1adb61f2b26da Nov 01 04:19:35.256748 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:35.256711 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" event={"ID":"d52635e3-4499-422b-a147-e66dbcc499f9","Type":"ContainerStarted","Data":"f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7"} Nov 01 04:19:35.256926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:35.256774 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" event={"ID":"d52635e3-4499-422b-a147-e66dbcc499f9","Type":"ContainerStarted","Data":"9d5e328712b0f4245f39ec1a9043a9b51ac6ab003c9f2b92a3a1adb61f2b26da"} Nov 01 04:19:35.256926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:35.256813 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:35.368461 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:35.368403 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" podStartSLOduration=1.368388145 podStartE2EDuration="1.368388145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:19:35.365180299 +0000 UTC m=+2186.954274929" watchObservedRunningTime="2026-11-01 04:19:35.368388145 +0000 UTC m=+2186.957482772" Nov 01 04:19:43.383541 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:43.383510 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls"] Nov 01 04:19:43.571546 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:43.571514 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b"] Nov 01 04:19:46.265116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.265081 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:19:46.720015 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.719972 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7"] Nov 01 04:19:46.725534 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.725509 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.777308 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.777211 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0ee0f8-serving-cert\"" Nov 01 04:19:46.795422 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.795388 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7"] Nov 01 04:19:46.800294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.800272 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.800393 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.800323 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.800436 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.800409 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrmp\" (UniqueName: \"kubernetes.io/projected/ea5764f5-962b-4b4e-9558-272684c3dd5d-kube-api-access-tjrmp\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.800666 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.800646 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5764f5-962b-4b4e-9558-272684c3dd5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.904065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.902775 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.904065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.902835 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrmp\" (UniqueName: \"kubernetes.io/projected/ea5764f5-962b-4b4e-9558-272684c3dd5d-kube-api-access-tjrmp\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.904065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.903016 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5764f5-962b-4b4e-9558-272684c3dd5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.904065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.903142 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.904065 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:19:46.903350 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-serving-cert: secret "maas-api-e2e-ipp-b-0ee0f8-serving-cert" not found Nov 01 04:19:46.904065 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:19:46.903424 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls podName:ea5764f5-962b-4b4e-9558-272684c3dd5d nodeName:}" failed. No retries permitted until 2026-11-01 04:19:47.403395784 +0000 UTC m=+2198.992490413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls") pod "maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" (UID: "ea5764f5-962b-4b4e-9558-272684c3dd5d") : secret "maas-api-e2e-ipp-b-0ee0f8-serving-cert" not found Nov 01 04:19:46.911698 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.911663 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.911853 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.911747 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5764f5-962b-4b4e-9558-272684c3dd5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:46.997006 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:46.996924 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrmp\" (UniqueName: \"kubernetes.io/projected/ea5764f5-962b-4b4e-9558-272684c3dd5d-kube-api-access-tjrmp\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:47.410611 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:47.410520 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:47.413081 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:47.413050 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:47.639254 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:47.639222 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:48.009337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.009307 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7"] Nov 01 04:19:48.011195 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:19:48.011167 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5764f5_962b_4b4e_9558_272684c3dd5d.slice/crio-15e982187b9f67a30f1b6e7e9bd154fdfa134810298f4ca7a86927bad03316ba WatchSource:0}: Error finding container 15e982187b9f67a30f1b6e7e9bd154fdfa134810298f4ca7a86927bad03316ba: Status 404 returned error can't find the container with id 15e982187b9f67a30f1b6e7e9bd154fdfa134810298f4ca7a86927bad03316ba Nov 01 04:19:48.340569 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.340482 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" event={"ID":"ea5764f5-962b-4b4e-9558-272684c3dd5d","Type":"ContainerStarted","Data":"15e982187b9f67a30f1b6e7e9bd154fdfa134810298f4ca7a86927bad03316ba"} Nov 01 04:19:48.384780 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.384746 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" podUID="b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" containerName="payload-processing" containerID="cri-o://3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2" gracePeriod=25 Nov 01 04:19:48.572459 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.572406 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" podUID="9d223186-7714-4b83-b5f1-d393853da167" containerName="payload-pre-processing" containerID="cri-o://53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755" gracePeriod=25 Nov 01 04:19:48.696451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.696430 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:19:48.829428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.829397 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-plugins-config-volume\") pod \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " Nov 01 04:19:48.829627 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.829529 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-kube-api-access-gmjdt\" (UniqueName: \"kubernetes.io/projected/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-kube-api-access-gmjdt\") pod \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\" (UID: \"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6\") " Nov 01 04:19:48.829981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.829929 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-plugins-config-volume" pod "b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" (UID: "b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:19:48.830181 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.830159 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:19:48.832531 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.832505 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-kube-api-access-gmjdt" pod "b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" (UID: "b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6"). InnerVolumeSpecName "kube-api-access-gmjdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:19:48.840607 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.840585 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:19:48.931147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.931123 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d223186-7714-4b83-b5f1-d393853da167-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d223186-7714-4b83-b5f1-d393853da167-plugins-config-volume\") pod \"9d223186-7714-4b83-b5f1-d393853da167\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " Nov 01 04:19:48.931253 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.931168 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d223186-7714-4b83-b5f1-d393853da167-kube-api-access-2fndv\" (UniqueName: \"kubernetes.io/projected/9d223186-7714-4b83-b5f1-d393853da167-kube-api-access-2fndv\") pod \"9d223186-7714-4b83-b5f1-d393853da167\" (UID: \"9d223186-7714-4b83-b5f1-d393853da167\") " Nov 01 04:19:48.931511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.931483 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d223186-7714-4b83-b5f1-d393853da167-plugins-config-volume" pod "9d223186-7714-4b83-b5f1-d393853da167" (UID: "9d223186-7714-4b83-b5f1-d393853da167"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:19:48.931719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.931703 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjdt\" (UniqueName: \"kubernetes.io/projected/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6-kube-api-access-gmjdt\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:19:48.931771 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.931721 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d223186-7714-4b83-b5f1-d393853da167-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:19:48.933313 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:48.933291 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d223186-7714-4b83-b5f1-d393853da167-kube-api-access-2fndv" pod "9d223186-7714-4b83-b5f1-d393853da167" (UID: "9d223186-7714-4b83-b5f1-d393853da167"). InnerVolumeSpecName "kube-api-access-2fndv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:19:49.032880 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.032847 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fndv\" (UniqueName: \"kubernetes.io/projected/9d223186-7714-4b83-b5f1-d393853da167-kube-api-access-2fndv\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:19:49.348434 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.348394 6735 generic.go:356] "Generic (PLEG): container finished" podID="b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" containerID="3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2" exitCode=0 Nov 01 04:19:49.348637 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.348458 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" Nov 01 04:19:49.348637 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.348453 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" event={"ID":"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6","Type":"ContainerDied","Data":"3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2"} Nov 01 04:19:49.348637 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.348581 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls" event={"ID":"b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6","Type":"ContainerDied","Data":"5f8e78e7e7d8659e1ce5b2c36629d0c13c4119a960cb632abf990270eb71087f"} Nov 01 04:19:49.348637 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.348629 6735 scope.go:122] "RemoveContainer" containerID="3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2" Nov 01 04:19:49.350123 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.350095 6735 generic.go:356] "Generic (PLEG): container finished" podID="9d223186-7714-4b83-b5f1-d393853da167" containerID="53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755" exitCode=0 Nov 01 04:19:49.350239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.350143 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" Nov 01 04:19:49.350239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.350169 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" event={"ID":"9d223186-7714-4b83-b5f1-d393853da167","Type":"ContainerDied","Data":"53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755"} Nov 01 04:19:49.350239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.350216 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b" event={"ID":"9d223186-7714-4b83-b5f1-d393853da167","Type":"ContainerDied","Data":"06479e8df0b5e93c4aaf295c8684d04565df68ce8f14165d55f1363e1b947d08"} Nov 01 04:19:49.352137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.352112 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" event={"ID":"ea5764f5-962b-4b4e-9558-272684c3dd5d","Type":"ContainerStarted","Data":"9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1"} Nov 01 04:19:49.352342 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.352318 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:19:49.360657 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.360630 6735 scope.go:122] "RemoveContainer" containerID="3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2" Nov 01 04:19:49.360929 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:19:49.360911 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2\": container with ID starting with 3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2 not found: ID does not exist" containerID="3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2" Nov 01 04:19:49.361190 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.360935 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2"} err="failed to get container status \"3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2\": rpc error: code = NotFound desc = could not find container \"3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2\": container with ID starting with 3697eca016b479d56956bae17f95370f2c322bab649a6fa7fa6b06a3755bc3b2 not found: ID does not exist" Nov 01 04:19:49.361190 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.360952 6735 scope.go:122] "RemoveContainer" containerID="53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755" Nov 01 04:19:49.369779 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.369764 6735 scope.go:122] "RemoveContainer" containerID="53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755" Nov 01 04:19:49.370049 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:19:49.370019 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755\": container with ID starting with 53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755 not found: ID does not exist" containerID="53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755" Nov 01 04:19:49.370121 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.370062 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755"} err="failed to get container status \"53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755\": rpc error: code = NotFound desc = could not find container \"53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755\": container with ID starting with 53525f8ee1f64f5b879a4314fb210563603fafa76974cb24493f88f16954a755 not found: ID does not exist" Nov 01 04:19:49.503443 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.503416 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b"] Nov 01 04:19:49.508057 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.508015 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a7b5f0-77d988f4c-pks7b"] Nov 01 04:19:49.615801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.615691 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" podStartSLOduration=2.725371171 podStartE2EDuration="3.615675026s" totalImagesPullingTime="890.303855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:19:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:19:49.614505966 +0000 UTC m=+2201.203600596" watchObservedRunningTime="2026-11-01 04:19:49.615675026 +0000 UTC m=+2201.204769657" Nov 01 04:19:49.718954 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.718913 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls"] Nov 01 04:19:49.723737 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:49.723702 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a7b5f0-59d45f45c7-bbvls"] Nov 01 04:19:50.529633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:50.529600 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d223186-7714-4b83-b5f1-d393853da167" path="/var/lib/kubelet/pods/9d223186-7714-4b83-b5f1-d393853da167/volumes" Nov 01 04:19:50.530064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:50.530049 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" path="/var/lib/kubelet/pods/b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6/volumes" Nov 01 04:19:55.365549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:19:55.365518 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:20:03.001821 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:20:03.001784 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:20:06.815328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:06.815293 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:20:33.000708 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:20:33.000674 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:20:34.266060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.265876 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk"] Nov 01 04:20:34.271510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.266915 6735 cpu_manager.go:414] "containerMap: removing container" podUID="9d223186-7714-4b83-b5f1-d393853da167" containerName="payload-pre-processing" Nov 01 04:20:34.271510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.266943 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d223186-7714-4b83-b5f1-d393853da167" containerName="payload-pre-processing" Nov 01 04:20:34.271510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.266961 6735 cpu_manager.go:414] "containerMap: removing container" podUID="b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" containerName="payload-processing" Nov 01 04:20:34.271510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.266969 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" containerName="payload-processing" Nov 01 04:20:34.271510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.267242 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d223186-7714-4b83-b5f1-d393853da167" containerName="payload-pre-processing" Nov 01 04:20:34.271510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.267258 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="b622d8c4-d5ce-4ec2-be3f-c8fd4a0fb9d6" containerName="payload-processing" Nov 01 04:20:34.271935 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.271766 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.295516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.295480 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd44cb06-9266-4b28-a59c-91294da97f1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.295680 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.295644 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8l7\" (UniqueName: \"kubernetes.io/projected/bd44cb06-9266-4b28-a59c-91294da97f1f-kube-api-access-5m8l7\") pod \"payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.316428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.316397 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk"] Nov 01 04:20:34.360064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.359998 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-862ad1\"" Nov 01 04:20:34.360064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.360044 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-862ad1-dockercfg-6v9lm\"" Nov 01 04:20:34.396456 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.396426 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd44cb06-9266-4b28-a59c-91294da97f1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.396668 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.396649 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8l7\" (UniqueName: \"kubernetes.io/projected/bd44cb06-9266-4b28-a59c-91294da97f1f-kube-api-access-5m8l7\") pod \"payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.397399 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.397371 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd44cb06-9266-4b28-a59c-91294da97f1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.454899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.454739 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8l7\" (UniqueName: \"kubernetes.io/projected/bd44cb06-9266-4b28-a59c-91294da97f1f-kube-api-access-5m8l7\") pod \"payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.598802 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.598720 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:34.971080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:34.971048 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk"] Nov 01 04:20:34.972676 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:20:34.972646 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd44cb06_9266_4b28_a59c_91294da97f1f.slice/crio-d528447032db1b8fcba943beea23945db63e6c13b20cc65bdb071911b059fcc5 WatchSource:0}: Error finding container d528447032db1b8fcba943beea23945db63e6c13b20cc65bdb071911b059fcc5: Status 404 returned error can't find the container with id d528447032db1b8fcba943beea23945db63e6c13b20cc65bdb071911b059fcc5 Nov 01 04:20:35.691641 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:35.691604 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" event={"ID":"bd44cb06-9266-4b28-a59c-91294da97f1f","Type":"ContainerStarted","Data":"2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78"} Nov 01 04:20:35.692063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:35.691662 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" event={"ID":"bd44cb06-9266-4b28-a59c-91294da97f1f","Type":"ContainerStarted","Data":"d528447032db1b8fcba943beea23945db63e6c13b20cc65bdb071911b059fcc5"} Nov 01 04:20:35.692063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:35.691737 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:20:35.801397 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:35.801342 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" podStartSLOduration=1.801327255 podStartE2EDuration="1.801327255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:20:35.800431064 +0000 UTC m=+2247.389525683" watchObservedRunningTime="2026-11-01 04:20:35.801327255 +0000 UTC m=+2247.390421890" Nov 01 04:20:46.700481 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:20:46.700448 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:21:00.604603 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.604568 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76"] Nov 01 04:21:00.608874 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.608854 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.624931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.624904 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/1ca55b28-a359-47b3-bcb3-a355eea4d24a-kube-api-access-zfmjm\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.625120 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.624998 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-metrics-certs\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.625204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.625188 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca55b28-a359-47b3-bcb3-a355eea4d24a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.625262 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.625216 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.657701 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.657671 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0882f4-serving-cert\"" Nov 01 04:21:00.707451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.707410 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76"] Nov 01 04:21:00.726690 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.726650 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/1ca55b28-a359-47b3-bcb3-a355eea4d24a-kube-api-access-zfmjm\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.726889 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.726722 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-metrics-certs\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.726889 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.726863 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca55b28-a359-47b3-bcb3-a355eea4d24a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.727007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.726896 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.727128 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:21:00.727105 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-serving-cert: secret "maas-api-e2e-ipp-b-0882f4-serving-cert" not found Nov 01 04:21:00.727213 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:21:00.727200 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls podName:1ca55b28-a359-47b3-bcb3-a355eea4d24a nodeName:}" failed. No retries permitted until 2026-11-01 04:21:01.227173897 +0000 UTC m=+2272.816268508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls") pod "maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" (UID: "1ca55b28-a359-47b3-bcb3-a355eea4d24a") : secret "maas-api-e2e-ipp-b-0882f4-serving-cert" not found Nov 01 04:21:00.727538 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.727519 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca55b28-a359-47b3-bcb3-a355eea4d24a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.729385 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.729361 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-metrics-certs\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:00.784814 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:00.784779 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/1ca55b28-a359-47b3-bcb3-a355eea4d24a-kube-api-access-zfmjm\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:01.233279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:01.233240 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:01.235884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:01.235855 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:01.523321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:01.523236 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:01.897814 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:01.897790 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76"] Nov 01 04:21:01.899773 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:21:01.899744 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca55b28_a359_47b3_bcb3_a355eea4d24a.slice/crio-8633e83b65ca0765c4e26d8dabec54a8cd08c3c62b83c74a5a8ec2abca6f5b68 WatchSource:0}: Error finding container 8633e83b65ca0765c4e26d8dabec54a8cd08c3c62b83c74a5a8ec2abca6f5b68: Status 404 returned error can't find the container with id 8633e83b65ca0765c4e26d8dabec54a8cd08c3c62b83c74a5a8ec2abca6f5b68 Nov 01 04:21:02.865890 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:02.865848 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" event={"ID":"1ca55b28-a359-47b3-bcb3-a355eea4d24a","Type":"ContainerStarted","Data":"8633e83b65ca0765c4e26d8dabec54a8cd08c3c62b83c74a5a8ec2abca6f5b68"} Nov 01 04:21:03.003202 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:21:03.003147 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:21:03.873573 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:03.873535 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" event={"ID":"1ca55b28-a359-47b3-bcb3-a355eea4d24a","Type":"ContainerStarted","Data":"f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474"} Nov 01 04:21:03.873758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:03.873599 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:03.986768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:03.986708 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" podStartSLOduration=2.985357184 podStartE2EDuration="3.986693136s" totalImagesPullingTime="1.001335952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:21:03.984676687 +0000 UTC m=+2275.573771319" watchObservedRunningTime="2026-11-01 04:21:03.986693136 +0000 UTC m=+2275.575787765" Nov 01 04:21:09.888638 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:09.888606 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:21:20.527202 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:20.527162 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:21:30.685179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:30.685151 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4qdkd_6dc0dfac-2281-4847-a3fe-2d11976315cc/payload-processing/0.log" Nov 01 04:21:30.942612 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:30.942530 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4qdkd_6dc0dfac-2281-4847-a3fe-2d11976315cc/payload-processing/0.log" Nov 01 04:21:33.002297 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:21:33.002262 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:21:53.693603 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:21:53.693560 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:22:03.002417 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:03.002379 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:22:15.634203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:15.634113 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv"] Nov 01 04:22:15.635008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:15.634969 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" podUID="9e3aec01-79aa-48a7-98be-34abc62658d6" containerName="payload-processing" containerID="cri-o://61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff" gracePeriod=30 Nov 01 04:22:15.691147 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:15.691108 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv"] Nov 01 04:22:15.691342 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:15.691321 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" podUID="5b94e72f-f79b-4be1-bd80-b017b412f0bc" containerName="payload-pre-processing" containerID="cri-o://1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6" gracePeriod=30 Nov 01 04:22:15.981467 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:15.981440 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:22:15.997979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:15.997955 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:22:16.025417 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.025385 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3aec01-79aa-48a7-98be-34abc62658d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3aec01-79aa-48a7-98be-34abc62658d6-plugins-config-volume\") pod \"9e3aec01-79aa-48a7-98be-34abc62658d6\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " Nov 01 04:22:16.025578 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.025484 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b94e72f-f79b-4be1-bd80-b017b412f0bc-kube-api-access-28czg\" (UniqueName: \"kubernetes.io/projected/5b94e72f-f79b-4be1-bd80-b017b412f0bc-kube-api-access-28czg\") pod \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " Nov 01 04:22:16.025631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.025616 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b94e72f-f79b-4be1-bd80-b017b412f0bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b94e72f-f79b-4be1-bd80-b017b412f0bc-plugins-config-volume\") pod \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\" (UID: \"5b94e72f-f79b-4be1-bd80-b017b412f0bc\") " Nov 01 04:22:16.025680 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.025639 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3aec01-79aa-48a7-98be-34abc62658d6-kube-api-access-cpbch\" (UniqueName: \"kubernetes.io/projected/9e3aec01-79aa-48a7-98be-34abc62658d6-kube-api-access-cpbch\") pod \"9e3aec01-79aa-48a7-98be-34abc62658d6\" (UID: \"9e3aec01-79aa-48a7-98be-34abc62658d6\") " Nov 01 04:22:16.025783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.025755 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3aec01-79aa-48a7-98be-34abc62658d6-plugins-config-volume" pod "9e3aec01-79aa-48a7-98be-34abc62658d6" (UID: "9e3aec01-79aa-48a7-98be-34abc62658d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:22:16.026007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.025985 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b94e72f-f79b-4be1-bd80-b017b412f0bc-plugins-config-volume" pod "5b94e72f-f79b-4be1-bd80-b017b412f0bc" (UID: "5b94e72f-f79b-4be1-bd80-b017b412f0bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:22:16.026241 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.026224 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e3aec01-79aa-48a7-98be-34abc62658d6-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:16.026292 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.026248 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b94e72f-f79b-4be1-bd80-b017b412f0bc-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:16.027916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.027890 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3aec01-79aa-48a7-98be-34abc62658d6-kube-api-access-cpbch" pod "9e3aec01-79aa-48a7-98be-34abc62658d6" (UID: "9e3aec01-79aa-48a7-98be-34abc62658d6"). InnerVolumeSpecName "kube-api-access-cpbch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:16.028020 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.027935 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b94e72f-f79b-4be1-bd80-b017b412f0bc-kube-api-access-28czg" pod "5b94e72f-f79b-4be1-bd80-b017b412f0bc" (UID: "5b94e72f-f79b-4be1-bd80-b017b412f0bc"). InnerVolumeSpecName "kube-api-access-28czg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:16.128027 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.127981 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpbch\" (UniqueName: \"kubernetes.io/projected/9e3aec01-79aa-48a7-98be-34abc62658d6-kube-api-access-cpbch\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:16.128027 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.128026 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28czg\" (UniqueName: \"kubernetes.io/projected/5b94e72f-f79b-4be1-bd80-b017b412f0bc-kube-api-access-28czg\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:16.344646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.344614 6735 generic.go:356] "Generic (PLEG): container finished" podID="5b94e72f-f79b-4be1-bd80-b017b412f0bc" containerID="1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6" exitCode=0 Nov 01 04:22:16.344845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.344674 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" Nov 01 04:22:16.344845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.344703 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" event={"ID":"5b94e72f-f79b-4be1-bd80-b017b412f0bc","Type":"ContainerDied","Data":"1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6"} Nov 01 04:22:16.344845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.344761 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv" event={"ID":"5b94e72f-f79b-4be1-bd80-b017b412f0bc","Type":"ContainerDied","Data":"aac1e50f3e37ded33372cd27b934227c58ae965446d53a7b3b2b98a53afb68b3"} Nov 01 04:22:16.344845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.344796 6735 scope.go:122] "RemoveContainer" containerID="1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6" Nov 01 04:22:16.346157 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.346125 6735 generic.go:356] "Generic (PLEG): container finished" podID="9e3aec01-79aa-48a7-98be-34abc62658d6" containerID="61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff" exitCode=0 Nov 01 04:22:16.346279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.346162 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" event={"ID":"9e3aec01-79aa-48a7-98be-34abc62658d6","Type":"ContainerDied","Data":"61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff"} Nov 01 04:22:16.346279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.346178 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" Nov 01 04:22:16.346279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.346205 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv" event={"ID":"9e3aec01-79aa-48a7-98be-34abc62658d6","Type":"ContainerDied","Data":"012c860cf80fb008059bfda91310e5117cb0bd86574e90b197b19963ba5d035e"} Nov 01 04:22:16.360443 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.360420 6735 scope.go:122] "RemoveContainer" containerID="1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6" Nov 01 04:22:16.360767 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:16.360741 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6\": container with ID starting with 1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6 not found: ID does not exist" containerID="1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6" Nov 01 04:22:16.360884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.360774 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6"} err="failed to get container status \"1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6\": rpc error: code = NotFound desc = could not find container \"1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6\": container with ID starting with 1a3b6b4b06012128346cc43487899d32f489bf3c39fb3f9879c056b5aeff3ce6 not found: ID does not exist" Nov 01 04:22:16.360884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.360794 6735 scope.go:122] "RemoveContainer" containerID="61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff" Nov 01 04:22:16.371014 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.370989 6735 scope.go:122] "RemoveContainer" containerID="61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff" Nov 01 04:22:16.371370 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:16.371350 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff\": container with ID starting with 61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff not found: ID does not exist" containerID="61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff" Nov 01 04:22:16.371443 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.371381 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff"} err="failed to get container status \"61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff\": rpc error: code = NotFound desc = could not find container \"61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff\": container with ID starting with 61d6d544613438fc116bffd03ff110ec86dae1a1613672b2e6339d94102e7bff not found: ID does not exist" Nov 01 04:22:16.499309 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.499269 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv"] Nov 01 04:22:16.502953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.502921 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-131718-549d9dc579-ks9kv"] Nov 01 04:22:16.527799 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.527765 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b94e72f-f79b-4be1-bd80-b017b412f0bc" path="/var/lib/kubelet/pods/5b94e72f-f79b-4be1-bd80-b017b412f0bc/volumes" Nov 01 04:22:16.606990 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.606901 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv"] Nov 01 04:22:16.611893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:16.611866 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-131718-5b57fc766d-mnzkv"] Nov 01 04:22:18.527499 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:18.527462 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3aec01-79aa-48a7-98be-34abc62658d6" path="/var/lib/kubelet/pods/9e3aec01-79aa-48a7-98be-34abc62658d6/volumes" Nov 01 04:22:26.887027 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:26.886981 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9"] Nov 01 04:22:26.887685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:26.887273 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" podUID="b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" containerName="maas-api" containerID="cri-o://246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43" gracePeriod=30 Nov 01 04:22:27.179395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.179366 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:22:27.286908 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.286876 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-kube-api-access-nvml8\" (UniqueName: \"kubernetes.io/projected/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-kube-api-access-nvml8\") pod \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " Nov 01 04:22:27.287130 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.287006 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls\") pod \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " Nov 01 04:22:27.287130 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.287030 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-metrics-certs\") pod \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " Nov 01 04:22:27.287232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.287220 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-openshift-service-ca\") pod \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\" (UID: \"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e\") " Nov 01 04:22:27.287655 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.287613 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-openshift-service-ca" pod "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" (UID: "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:22:27.289450 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.289407 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-metrics-certs" pod "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" (UID: "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:22:27.289808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.289721 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-kube-api-access-nvml8" pod "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" (UID: "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e"). InnerVolumeSpecName "kube-api-access-nvml8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:27.289907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.289880 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls" pod "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" (UID: "b6ab6e5f-8227-44f6-84e3-921ef42ffe3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:22:27.388788 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.388749 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:27.388788 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.388780 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvml8\" (UniqueName: \"kubernetes.io/projected/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-kube-api-access-nvml8\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:27.388788 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.388790 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:27.388788 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.388798 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:27.424018 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.423921 6735 generic.go:356] "Generic (PLEG): container finished" podID="b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" containerID="246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43" exitCode=0 Nov 01 04:22:27.424018 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.423971 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" Nov 01 04:22:27.424262 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.424019 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" event={"ID":"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e","Type":"ContainerDied","Data":"246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43"} Nov 01 04:22:27.424262 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.424082 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9" event={"ID":"b6ab6e5f-8227-44f6-84e3-921ef42ffe3e","Type":"ContainerDied","Data":"bde2b673ca4669f391c8d6c781aa25daad5cb6170204ffb03c1e46d39278bcca"} Nov 01 04:22:27.424262 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.424109 6735 scope.go:122] "RemoveContainer" containerID="246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43" Nov 01 04:22:27.441406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.441373 6735 scope.go:122] "RemoveContainer" containerID="246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43" Nov 01 04:22:27.441732 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:27.441702 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43\": container with ID starting with 246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43 not found: ID does not exist" containerID="246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43" Nov 01 04:22:27.441798 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.441734 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43"} err="failed to get container status \"246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43\": rpc error: code = NotFound desc = could not find container \"246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43\": container with ID starting with 246ccc523ab6cb67c61585d11a950c2659a3858acf66481dceb6764fb333ef43 not found: ID does not exist" Nov 01 04:22:27.466409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.466374 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh"] Nov 01 04:22:27.466628 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.466602 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" podUID="d52635e3-4499-422b-a147-e66dbcc499f9" containerName="payload-pre-processing" containerID="cri-o://f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7" gracePeriod=30 Nov 01 04:22:27.591997 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.591964 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9"] Nov 01 04:22:27.593638 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.593332 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-601a4b-769b76dc4b-llpg9"] Nov 01 04:22:27.811900 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.811812 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:22:27.906914 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.906396 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52635e3-4499-422b-a147-e66dbcc499f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52635e3-4499-422b-a147-e66dbcc499f9-plugins-config-volume\") pod \"d52635e3-4499-422b-a147-e66dbcc499f9\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " Nov 01 04:22:27.907630 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.906858 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52635e3-4499-422b-a147-e66dbcc499f9-plugins-config-volume" pod "d52635e3-4499-422b-a147-e66dbcc499f9" (UID: "d52635e3-4499-422b-a147-e66dbcc499f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:22:27.907630 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.907014 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52635e3-4499-422b-a147-e66dbcc499f9-kube-api-access-6kwpt\" (UniqueName: \"kubernetes.io/projected/d52635e3-4499-422b-a147-e66dbcc499f9-kube-api-access-6kwpt\") pod \"d52635e3-4499-422b-a147-e66dbcc499f9\" (UID: \"d52635e3-4499-422b-a147-e66dbcc499f9\") " Nov 01 04:22:27.908273 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.908178 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52635e3-4499-422b-a147-e66dbcc499f9-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:27.915976 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:27.915946 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52635e3-4499-422b-a147-e66dbcc499f9-kube-api-access-6kwpt" pod "d52635e3-4499-422b-a147-e66dbcc499f9" (UID: "d52635e3-4499-422b-a147-e66dbcc499f9"). InnerVolumeSpecName "kube-api-access-6kwpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:28.009244 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.009142 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwpt\" (UniqueName: \"kubernetes.io/projected/d52635e3-4499-422b-a147-e66dbcc499f9-kube-api-access-6kwpt\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:28.431609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.431514 6735 generic.go:356] "Generic (PLEG): container finished" podID="d52635e3-4499-422b-a147-e66dbcc499f9" containerID="f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7" exitCode=0 Nov 01 04:22:28.431795 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.431632 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" Nov 01 04:22:28.431964 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.431663 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" event={"ID":"d52635e3-4499-422b-a147-e66dbcc499f9","Type":"ContainerDied","Data":"f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7"} Nov 01 04:22:28.432072 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.432011 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh" event={"ID":"d52635e3-4499-422b-a147-e66dbcc499f9","Type":"ContainerDied","Data":"9d5e328712b0f4245f39ec1a9043a9b51ac6ab003c9f2b92a3a1adb61f2b26da"} Nov 01 04:22:28.432072 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.432065 6735 scope.go:122] "RemoveContainer" containerID="f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7" Nov 01 04:22:28.443000 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.442981 6735 scope.go:122] "RemoveContainer" containerID="f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7" Nov 01 04:22:28.443367 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:28.443305 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7\": container with ID starting with f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7 not found: ID does not exist" containerID="f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7" Nov 01 04:22:28.443417 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.443377 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7"} err="failed to get container status \"f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7\": rpc error: code = NotFound desc = could not find container \"f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7\": container with ID starting with f5df097f597f331c1b338c513a28bfc14496f4ee6b14ae049bad3ea64c4b2ec7 not found: ID does not exist" Nov 01 04:22:28.526541 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.526499 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" path="/var/lib/kubelet/pods/b6ab6e5f-8227-44f6-84e3-921ef42ffe3e/volumes" Nov 01 04:22:28.586931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.586895 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh"] Nov 01 04:22:28.590633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:28.590608 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-601a4b-64569c499f-wchrh"] Nov 01 04:22:29.031372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:29.031342 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:22:30.538949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:30.538912 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52635e3-4499-422b-a147-e66dbcc499f9" path="/var/lib/kubelet/pods/d52635e3-4499-422b-a147-e66dbcc499f9/volumes" Nov 01 04:22:33.001806 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:33.001770 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:22:38.741367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:38.741335 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6"] Nov 01 04:22:38.741803 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:38.741546 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" containerID="cri-o://024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d" gracePeriod=30 Nov 01 04:22:41.515971 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:41.515932 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr"] Nov 01 04:22:41.516475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:41.516210 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" containerID="cri-o://d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011" gracePeriod=30 Nov 01 04:22:44.012443 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.012367 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:22:44.187181 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187142 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6a15781-c068-4965-87da-93bbaa57cc01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a15781-c068-4965-87da-93bbaa57cc01-istio-podinfo\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187386 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187188 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-istio-token\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-istio-token\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187386 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187230 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-data\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187386 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187267 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-socket\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187386 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187306 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-certs\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187386 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187361 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a15781-c068-4965-87da-93bbaa57cc01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a15781-c068-4965-87da-93bbaa57cc01-istiod-ca-cert\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187436 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-credential-socket\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187483 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-envoy\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187562 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-kube-api-access-hfqt7\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-kube-api-access-hfqt7\") pod \"c6a15781-c068-4965-87da-93bbaa57cc01\" (UID: \"c6a15781-c068-4965-87da-93bbaa57cc01\") " Nov 01 04:22:44.187794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187690 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-certs" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:44.187919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187864 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-credential-socket" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:44.187919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.187880 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a15781-c068-4965-87da-93bbaa57cc01-istiod-ca-cert" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:22:44.188203 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.188174 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-socket" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:44.188331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.188306 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.188416 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.188376 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.188416 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.188394 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6a15781-c068-4965-87da-93bbaa57cc01-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.188416 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.188409 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.188558 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.188421 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-data" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:44.189641 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.189618 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6a15781-c068-4965-87da-93bbaa57cc01-istio-podinfo" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:22:44.189928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.189912 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-kube-api-access-hfqt7" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "kube-api-access-hfqt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:44.190288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.190260 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-envoy" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:44.190288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.190279 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-istio-token" pod "c6a15781-c068-4965-87da-93bbaa57cc01" (UID: "c6a15781-c068-4965-87da-93bbaa57cc01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:44.289427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.289341 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.289427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.289372 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqt7\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-kube-api-access-hfqt7\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.289427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.289382 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6a15781-c068-4965-87da-93bbaa57cc01-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.289427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.289390 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6a15781-c068-4965-87da-93bbaa57cc01-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.289427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.289399 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6a15781-c068-4965-87da-93bbaa57cc01-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:44.536126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.536065 6735 generic.go:356] "Generic (PLEG): container finished" podID="c6a15781-c068-4965-87da-93bbaa57cc01" containerID="024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d" exitCode=0 Nov 01 04:22:44.536126 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.536112 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" Nov 01 04:22:44.536269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.536164 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" event={"ID":"c6a15781-c068-4965-87da-93bbaa57cc01","Type":"ContainerDied","Data":"024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d"} Nov 01 04:22:44.536269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.536228 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6" event={"ID":"c6a15781-c068-4965-87da-93bbaa57cc01","Type":"ContainerDied","Data":"c9699ca085d3b999a91490fd1520be2d7f019f6c27a75a938c70ee7dc3f8d279"} Nov 01 04:22:44.536269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.536261 6735 scope.go:122] "RemoveContainer" containerID="024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d" Nov 01 04:22:44.546459 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.546439 6735 scope.go:122] "RemoveContainer" containerID="024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d" Nov 01 04:22:44.546785 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:44.546752 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d\": container with ID starting with 024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d not found: ID does not exist" containerID="024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d" Nov 01 04:22:44.546907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.546794 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d"} err="failed to get container status \"024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d\": rpc error: code = NotFound desc = could not find container \"024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d\": container with ID starting with 024d23134e4b413ec70d6ff25ad6ffd9df443391fbb676e6580a297838532c9d not found: ID does not exist" Nov 01 04:22:44.692281 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.692247 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6"] Nov 01 04:22:44.738625 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:44.738583 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2c259-openshift-default-6465f87cb7-v88r6"] Nov 01 04:22:45.175194 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:45.175162 6735 patch_prober.go:29] interesting pod/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:22:45.175586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:45.175234 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:22:46.528540 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.528497 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" path="/var/lib/kubelet/pods/c6a15781-c068-4965-87da-93bbaa57cc01/volumes" Nov 01 04:22:46.789122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.789023 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 04:22:46.921914 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.921880 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-credential-socket\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.921932 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc8b0b4-6032-451e-9cc8-0e666197d896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8b0b4-6032-451e-9cc8-0e666197d896-istiod-ca-cert\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.921964 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-envoy\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.921988 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-socket\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922027 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-token\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922116 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-data\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922193 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-certs\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922212 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-credential-socket" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:46.922449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922230 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-podinfo\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922283 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-kube-api-access-fc6sh\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-kube-api-access-fc6sh\") pod \"fcc8b0b4-6032-451e-9cc8-0e666197d896\" (UID: \"fcc8b0b4-6032-451e-9cc8-0e666197d896\") " Nov 01 04:22:46.922449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922397 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-socket" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:46.922449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922426 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-data" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:46.922756 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922485 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc8b0b4-6032-451e-9cc8-0e666197d896-istiod-ca-cert" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:22:46.922756 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922591 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-certs" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:46.923002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922939 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:46.923002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922962 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:46.923002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922978 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8b0b4-6032-451e-9cc8-0e666197d896-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:46.923002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.922992 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:46.923002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.923006 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:46.924633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.924596 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-envoy" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:22:46.924751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.924702 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-token" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:46.925145 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.925127 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-podinfo" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:22:46.925145 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:46.925134 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-kube-api-access-fc6sh" pod "fcc8b0b4-6032-451e-9cc8-0e666197d896" (UID: "fcc8b0b4-6032-451e-9cc8-0e666197d896"). InnerVolumeSpecName "kube-api-access-fc6sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:22:47.025764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.025720 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:47.025764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.025764 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:47.026072 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.025790 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc8b0b4-6032-451e-9cc8-0e666197d896-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:47.026072 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.025806 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc6sh\" (UniqueName: \"kubernetes.io/projected/fcc8b0b4-6032-451e-9cc8-0e666197d896-kube-api-access-fc6sh\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:22:47.557418 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.557383 6735 generic.go:356] "Generic (PLEG): container finished" podID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerID="d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011" exitCode=0 Nov 01 04:22:47.557875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.557443 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" Nov 01 04:22:47.557875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.557468 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" event={"ID":"fcc8b0b4-6032-451e-9cc8-0e666197d896","Type":"ContainerDied","Data":"d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011"} Nov 01 04:22:47.557875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.557516 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr" event={"ID":"fcc8b0b4-6032-451e-9cc8-0e666197d896","Type":"ContainerDied","Data":"4838015b3e0c599fcd17eff2f892259a37c5e22182188cf36641917a1a09b425"} Nov 01 04:22:47.557875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.557542 6735 scope.go:122] "RemoveContainer" containerID="d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011" Nov 01 04:22:47.568392 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.568375 6735 scope.go:122] "RemoveContainer" containerID="265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e" Nov 01 04:22:47.577625 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.577603 6735 scope.go:122] "RemoveContainer" containerID="d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011" Nov 01 04:22:47.577944 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:47.577922 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011\": container with ID starting with d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011 not found: ID does not exist" containerID="d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011" Nov 01 04:22:47.578029 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.577956 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011"} err="failed to get container status \"d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011\": rpc error: code = NotFound desc = could not find container \"d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011\": container with ID starting with d0a609f1a45b7c7bbba8fe96faa927800847857d47aefdda53ccf2449b28d011 not found: ID does not exist" Nov 01 04:22:47.578029 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.577980 6735 scope.go:122] "RemoveContainer" containerID="265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e" Nov 01 04:22:47.578257 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:22:47.578239 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e\": container with ID starting with 265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e not found: ID does not exist" containerID="265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e" Nov 01 04:22:47.578316 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.578264 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e"} err="failed to get container status \"265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e\": rpc error: code = NotFound desc = could not find container \"265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e\": container with ID starting with 265868d11999ce133651c53cd1b260aab0de943bce0769ebabe1c4181e9ee01e not found: ID does not exist" Nov 01 04:22:47.715083 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.715028 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr"] Nov 01 04:22:47.761585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:47.761550 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a9c5b3-openshift-default-594d45cd9f-c8jlr"] Nov 01 04:22:48.526993 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:22:48.526957 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" path="/var/lib/kubelet/pods/fcc8b0b4-6032-451e-9cc8-0e666197d896/volumes" Nov 01 04:23:01.290800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:01.290767 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:23:03.001797 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:03.001766 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:23:04.788415 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:04.788382 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb"] Nov 01 04:23:04.788884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:04.788608 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" containerID="cri-o://40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851" gracePeriod=30 Nov 01 04:23:09.828228 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:09.828202 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:23:09.834238 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:09.834221 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:23:10.051538 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.051511 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:23:10.206494 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206404 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-podinfo\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.206494 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206455 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-kube-api-access-pg2sh\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-kube-api-access-pg2sh\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.206494 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206481 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-data\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.206791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206559 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-envoy\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.206791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206604 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-token\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-token\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.206791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206634 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-socket\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.206791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206713 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istiod-ca-cert\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.206791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206783 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-certs\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.207155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206817 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-data" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:23:10.207155 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.206847 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-credential-socket\") pod \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\" (UID: \"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e\") " Nov 01 04:23:10.207434 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207157 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istiod-ca-cert" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:10.207434 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207240 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-certs" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:23:10.207549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207464 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-socket" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:23:10.207612 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207539 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-credential-socket" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:23:10.207776 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207755 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.207847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207782 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.207847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207797 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.207847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207813 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.207847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.207827 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.209078 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.209022 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-token" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:10.209215 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.209080 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-envoy" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:23:10.209215 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.209093 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-podinfo" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:23:10.209215 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.209130 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-kube-api-access-pg2sh" pod "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" (UID: "fdf893cd-f7ae-43fc-b66f-0840ad6cc48e"). InnerVolumeSpecName "kube-api-access-pg2sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:10.309001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.308969 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.309001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.308995 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.309001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.309005 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2sh\" (UniqueName: \"kubernetes.io/projected/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-kube-api-access-pg2sh\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.309268 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.309015 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:10.699461 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.699423 6735 generic.go:356] "Generic (PLEG): container finished" podID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerID="40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851" exitCode=0 Nov 01 04:23:10.699641 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.699510 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" Nov 01 04:23:10.699641 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.699509 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" event={"ID":"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e","Type":"ContainerDied","Data":"40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851"} Nov 01 04:23:10.699730 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.699648 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb" event={"ID":"fdf893cd-f7ae-43fc-b66f-0840ad6cc48e","Type":"ContainerDied","Data":"a95567e41cb7f9e46d92059f929ee8d8d8b285bb5878c08afdb20c3bcd86030f"} Nov 01 04:23:10.699730 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.699700 6735 scope.go:122] "RemoveContainer" containerID="40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851" Nov 01 04:23:10.709679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.709661 6735 scope.go:122] "RemoveContainer" containerID="40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851" Nov 01 04:23:10.709971 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:10.709951 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851\": container with ID starting with 40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851 not found: ID does not exist" containerID="40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851" Nov 01 04:23:10.710017 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.709980 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851"} err="failed to get container status \"40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851\": rpc error: code = NotFound desc = could not find container \"40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851\": container with ID starting with 40ed928eb201695a412ca10ab3c876f0502526bbb30dc54305febea5f03cc851 not found: ID does not exist" Nov 01 04:23:10.854707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.854629 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb"] Nov 01 04:23:10.858624 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:10.858597 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zhpnb"] Nov 01 04:23:12.526339 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:12.526306 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" path="/var/lib/kubelet/pods/fdf893cd-f7ae-43fc-b66f-0840ad6cc48e/volumes" Nov 01 04:23:16.162121 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:16.162086 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd"] Nov 01 04:23:16.262780 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:16.262748 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx"] Nov 01 04:23:21.163080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.163005 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" podUID="c9c74d9c-c29c-4943-adaf-34dc271e5afc" containerName="payload-processing" containerID="cri-o://86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8" gracePeriod=25 Nov 01 04:23:21.263324 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.263274 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" podUID="b4e528ab-13d2-4e11-bd51-bb2141f465bf" containerName="payload-pre-processing" containerID="cri-o://a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a" gracePeriod=25 Nov 01 04:23:21.447772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.447743 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:23:21.526209 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.526178 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:23:21.561575 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.561542 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c74d9c-c29c-4943-adaf-34dc271e5afc-kube-api-access-4rnd7\" (UniqueName: \"kubernetes.io/projected/c9c74d9c-c29c-4943-adaf-34dc271e5afc-kube-api-access-4rnd7\") pod \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " Nov 01 04:23:21.561745 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.561667 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c74d9c-c29c-4943-adaf-34dc271e5afc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74d9c-c29c-4943-adaf-34dc271e5afc-plugins-config-volume\") pod \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\" (UID: \"c9c74d9c-c29c-4943-adaf-34dc271e5afc\") " Nov 01 04:23:21.561999 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.561979 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c74d9c-c29c-4943-adaf-34dc271e5afc-plugins-config-volume" pod "c9c74d9c-c29c-4943-adaf-34dc271e5afc" (UID: "c9c74d9c-c29c-4943-adaf-34dc271e5afc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:21.564166 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.564143 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c74d9c-c29c-4943-adaf-34dc271e5afc-kube-api-access-4rnd7" pod "c9c74d9c-c29c-4943-adaf-34dc271e5afc" (UID: "c9c74d9c-c29c-4943-adaf-34dc271e5afc"). InnerVolumeSpecName "kube-api-access-4rnd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:21.662856 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.662826 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e528ab-13d2-4e11-bd51-bb2141f465bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e528ab-13d2-4e11-bd51-bb2141f465bf-plugins-config-volume\") pod \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " Nov 01 04:23:21.663054 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.662875 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e528ab-13d2-4e11-bd51-bb2141f465bf-kube-api-access-8kvcl\" (UniqueName: \"kubernetes.io/projected/b4e528ab-13d2-4e11-bd51-bb2141f465bf-kube-api-access-8kvcl\") pod \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\" (UID: \"b4e528ab-13d2-4e11-bd51-bb2141f465bf\") " Nov 01 04:23:21.663233 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.663212 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e528ab-13d2-4e11-bd51-bb2141f465bf-plugins-config-volume" pod "b4e528ab-13d2-4e11-bd51-bb2141f465bf" (UID: "b4e528ab-13d2-4e11-bd51-bb2141f465bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:21.663325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.663311 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rnd7\" (UniqueName: \"kubernetes.io/projected/c9c74d9c-c29c-4943-adaf-34dc271e5afc-kube-api-access-4rnd7\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:21.663369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.663327 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e528ab-13d2-4e11-bd51-bb2141f465bf-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:21.663369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.663336 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c74d9c-c29c-4943-adaf-34dc271e5afc-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:21.665133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.665105 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e528ab-13d2-4e11-bd51-bb2141f465bf-kube-api-access-8kvcl" pod "b4e528ab-13d2-4e11-bd51-bb2141f465bf" (UID: "b4e528ab-13d2-4e11-bd51-bb2141f465bf"). InnerVolumeSpecName "kube-api-access-8kvcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:21.764493 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.764459 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvcl\" (UniqueName: \"kubernetes.io/projected/b4e528ab-13d2-4e11-bd51-bb2141f465bf-kube-api-access-8kvcl\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:21.778343 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.778305 6735 generic.go:356] "Generic (PLEG): container finished" podID="b4e528ab-13d2-4e11-bd51-bb2141f465bf" containerID="a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a" exitCode=0 Nov 01 04:23:21.778549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.778348 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" Nov 01 04:23:21.778549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.778389 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" event={"ID":"b4e528ab-13d2-4e11-bd51-bb2141f465bf","Type":"ContainerDied","Data":"a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a"} Nov 01 04:23:21.778549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.778458 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx" event={"ID":"b4e528ab-13d2-4e11-bd51-bb2141f465bf","Type":"ContainerDied","Data":"75ebf5dcf7e4a01b125bc65daa8ec825cf3016f375f6704141608f0fd00b92c5"} Nov 01 04:23:21.778549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.778489 6735 scope.go:122] "RemoveContainer" containerID="a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a" Nov 01 04:23:21.780019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.779994 6735 generic.go:356] "Generic (PLEG): container finished" podID="c9c74d9c-c29c-4943-adaf-34dc271e5afc" containerID="86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8" exitCode=0 Nov 01 04:23:21.780324 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.780095 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" Nov 01 04:23:21.780324 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.780107 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" event={"ID":"c9c74d9c-c29c-4943-adaf-34dc271e5afc","Type":"ContainerDied","Data":"86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8"} Nov 01 04:23:21.780324 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.780153 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd" event={"ID":"c9c74d9c-c29c-4943-adaf-34dc271e5afc","Type":"ContainerDied","Data":"d8cc98f764a858bc6a1d13d2012bfad4ae96e8e580d76c5b3ffd3c2fb69b631b"} Nov 01 04:23:21.788695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.788675 6735 scope.go:122] "RemoveContainer" containerID="a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a" Nov 01 04:23:21.788992 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:21.788951 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a\": container with ID starting with a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a not found: ID does not exist" containerID="a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a" Nov 01 04:23:21.789289 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.788987 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a"} err="failed to get container status \"a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a\": rpc error: code = NotFound desc = could not find container \"a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a\": container with ID starting with a29b0c6814c8c2f62f3035a5f0bcbcc28b7234dd5c9cfdff7492c1b2d3c84a4a not found: ID does not exist" Nov 01 04:23:21.789289 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.789024 6735 scope.go:122] "RemoveContainer" containerID="86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8" Nov 01 04:23:21.798110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.798092 6735 scope.go:122] "RemoveContainer" containerID="86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8" Nov 01 04:23:21.798380 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:21.798362 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8\": container with ID starting with 86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8 not found: ID does not exist" containerID="86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8" Nov 01 04:23:21.798438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.798386 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8"} err="failed to get container status \"86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8\": rpc error: code = NotFound desc = could not find container \"86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8\": container with ID starting with 86bd5c93608e21d886fb5e1d554fd55f0ebdd5899dd08b0edbf2165c1a7823b8 not found: ID does not exist" Nov 01 04:23:21.933578 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.933545 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd"] Nov 01 04:23:21.937847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:21.937820 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4c2a66-556f69bb59-kdwrd"] Nov 01 04:23:22.040275 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:22.040195 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx"] Nov 01 04:23:22.043991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:22.043968 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4c2a66-54d499db4c-z5wnx"] Nov 01 04:23:22.527437 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:22.527405 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e528ab-13d2-4e11-bd51-bb2141f465bf" path="/var/lib/kubelet/pods/b4e528ab-13d2-4e11-bd51-bb2141f465bf/volumes" Nov 01 04:23:22.527821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:22.527807 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c74d9c-c29c-4943-adaf-34dc271e5afc" path="/var/lib/kubelet/pods/c9c74d9c-c29c-4943-adaf-34dc271e5afc/volumes" Nov 01 04:23:33.001990 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:33.001942 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:23:38.642077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.641992 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd"] Nov 01 04:23:38.645743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.645682 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" podUID="89975435-db64-486e-bd27-b603abc3a312" containerName="maas-api" containerID="cri-o://89be19d9a55cf490e1f45509d2a1b83f9f94276e6d2de5ac6eee048b1b8b7e9d" gracePeriod=30 Nov 01 04:23:38.905146 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.905054 6735 generic.go:356] "Generic (PLEG): container finished" podID="89975435-db64-486e-bd27-b603abc3a312" containerID="89be19d9a55cf490e1f45509d2a1b83f9f94276e6d2de5ac6eee048b1b8b7e9d" exitCode=0 Nov 01 04:23:38.905146 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.905069 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" event={"ID":"89975435-db64-486e-bd27-b603abc3a312","Type":"ContainerDied","Data":"89be19d9a55cf490e1f45509d2a1b83f9f94276e6d2de5ac6eee048b1b8b7e9d"} Nov 01 04:23:38.950262 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.950238 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:23:38.988293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.988265 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-metrics-certs\") pod \"89975435-db64-486e-bd27-b603abc3a312\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " Nov 01 04:23:38.988453 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.988315 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89975435-db64-486e-bd27-b603abc3a312-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89975435-db64-486e-bd27-b603abc3a312-openshift-service-ca\") pod \"89975435-db64-486e-bd27-b603abc3a312\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " Nov 01 04:23:38.988453 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.988436 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls\") pod \"89975435-db64-486e-bd27-b603abc3a312\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " Nov 01 04:23:38.988549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.988467 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89975435-db64-486e-bd27-b603abc3a312-kube-api-access-dvg5g\" (UniqueName: \"kubernetes.io/projected/89975435-db64-486e-bd27-b603abc3a312-kube-api-access-dvg5g\") pod \"89975435-db64-486e-bd27-b603abc3a312\" (UID: \"89975435-db64-486e-bd27-b603abc3a312\") " Nov 01 04:23:38.988735 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.988701 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89975435-db64-486e-bd27-b603abc3a312-openshift-service-ca" pod "89975435-db64-486e-bd27-b603abc3a312" (UID: "89975435-db64-486e-bd27-b603abc3a312"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:38.990666 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.990637 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls" pod "89975435-db64-486e-bd27-b603abc3a312" (UID: "89975435-db64-486e-bd27-b603abc3a312"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:23:38.990771 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.990705 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89975435-db64-486e-bd27-b603abc3a312-kube-api-access-dvg5g" pod "89975435-db64-486e-bd27-b603abc3a312" (UID: "89975435-db64-486e-bd27-b603abc3a312"). InnerVolumeSpecName "kube-api-access-dvg5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:38.990771 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:38.990741 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-metrics-certs" pod "89975435-db64-486e-bd27-b603abc3a312" (UID: "89975435-db64-486e-bd27-b603abc3a312"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:23:39.090178 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.090142 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89975435-db64-486e-bd27-b603abc3a312-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.090178 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.090181 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.090382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.090197 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvg5g\" (UniqueName: \"kubernetes.io/projected/89975435-db64-486e-bd27-b603abc3a312-kube-api-access-dvg5g\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.090382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.090214 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89975435-db64-486e-bd27-b603abc3a312-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.093623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.093592 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67"] Nov 01 04:23:39.093857 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.093833 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" podUID="426b6f1e-f328-405c-86b7-00f2152e5567" containerName="payload-processing" containerID="cri-o://c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc" gracePeriod=30 Nov 01 04:23:39.152018 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.151982 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m"] Nov 01 04:23:39.152279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.152256 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" podUID="dacc0e05-a8f1-414c-b12a-deed8686bf07" containerName="payload-pre-processing" containerID="cri-o://4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce" gracePeriod=30 Nov 01 04:23:39.397855 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.397827 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:23:39.433065 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.433011 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:23:39.494332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494301 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426b6f1e-f328-405c-86b7-00f2152e5567-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b6f1e-f328-405c-86b7-00f2152e5567-plugins-config-volume\") pod \"426b6f1e-f328-405c-86b7-00f2152e5567\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " Nov 01 04:23:39.494332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494338 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacc0e05-a8f1-414c-b12a-deed8686bf07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc0e05-a8f1-414c-b12a-deed8686bf07-plugins-config-volume\") pod \"dacc0e05-a8f1-414c-b12a-deed8686bf07\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " Nov 01 04:23:39.494533 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494378 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacc0e05-a8f1-414c-b12a-deed8686bf07-kube-api-access-76v2p\" (UniqueName: \"kubernetes.io/projected/dacc0e05-a8f1-414c-b12a-deed8686bf07-kube-api-access-76v2p\") pod \"dacc0e05-a8f1-414c-b12a-deed8686bf07\" (UID: \"dacc0e05-a8f1-414c-b12a-deed8686bf07\") " Nov 01 04:23:39.494644 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494624 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426b6f1e-f328-405c-86b7-00f2152e5567-kube-api-access-jc4hs\" (UniqueName: \"kubernetes.io/projected/426b6f1e-f328-405c-86b7-00f2152e5567-kube-api-access-jc4hs\") pod \"426b6f1e-f328-405c-86b7-00f2152e5567\" (UID: \"426b6f1e-f328-405c-86b7-00f2152e5567\") " Nov 01 04:23:39.494737 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494715 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacc0e05-a8f1-414c-b12a-deed8686bf07-plugins-config-volume" pod "dacc0e05-a8f1-414c-b12a-deed8686bf07" (UID: "dacc0e05-a8f1-414c-b12a-deed8686bf07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:39.494737 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494724 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426b6f1e-f328-405c-86b7-00f2152e5567-plugins-config-volume" pod "426b6f1e-f328-405c-86b7-00f2152e5567" (UID: "426b6f1e-f328-405c-86b7-00f2152e5567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:39.494984 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494966 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/426b6f1e-f328-405c-86b7-00f2152e5567-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.494984 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.494984 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacc0e05-a8f1-414c-b12a-deed8686bf07-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.496659 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.496637 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacc0e05-a8f1-414c-b12a-deed8686bf07-kube-api-access-76v2p" pod "dacc0e05-a8f1-414c-b12a-deed8686bf07" (UID: "dacc0e05-a8f1-414c-b12a-deed8686bf07"). InnerVolumeSpecName "kube-api-access-76v2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:39.496764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.496665 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426b6f1e-f328-405c-86b7-00f2152e5567-kube-api-access-jc4hs" pod "426b6f1e-f328-405c-86b7-00f2152e5567" (UID: "426b6f1e-f328-405c-86b7-00f2152e5567"). InnerVolumeSpecName "kube-api-access-jc4hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:39.595975 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.595941 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc4hs\" (UniqueName: \"kubernetes.io/projected/426b6f1e-f328-405c-86b7-00f2152e5567-kube-api-access-jc4hs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.595975 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.595973 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76v2p\" (UniqueName: \"kubernetes.io/projected/dacc0e05-a8f1-414c-b12a-deed8686bf07-kube-api-access-76v2p\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:39.912182 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.912149 6735 generic.go:356] "Generic (PLEG): container finished" podID="dacc0e05-a8f1-414c-b12a-deed8686bf07" containerID="4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce" exitCode=0 Nov 01 04:23:39.912595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.912206 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" Nov 01 04:23:39.912595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.912230 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" event={"ID":"dacc0e05-a8f1-414c-b12a-deed8686bf07","Type":"ContainerDied","Data":"4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce"} Nov 01 04:23:39.912595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.912307 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m" event={"ID":"dacc0e05-a8f1-414c-b12a-deed8686bf07","Type":"ContainerDied","Data":"5d63e7c133aed4da202cd30076732d0fd6c4eda46cce6f049110fa90a8fd75a1"} Nov 01 04:23:39.912595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.912334 6735 scope.go:122] "RemoveContainer" containerID="4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce" Nov 01 04:23:39.913839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.913819 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" Nov 01 04:23:39.913970 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.913833 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd" event={"ID":"89975435-db64-486e-bd27-b603abc3a312","Type":"ContainerDied","Data":"44e553994984aac3adbac0780ba931574975cc16162e6e50459ac659c7bad4fa"} Nov 01 04:23:39.915519 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.915474 6735 generic.go:356] "Generic (PLEG): container finished" podID="426b6f1e-f328-405c-86b7-00f2152e5567" containerID="c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc" exitCode=0 Nov 01 04:23:39.915643 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.915526 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" Nov 01 04:23:39.915643 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.915573 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" event={"ID":"426b6f1e-f328-405c-86b7-00f2152e5567","Type":"ContainerDied","Data":"c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc"} Nov 01 04:23:39.915643 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.915631 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67" event={"ID":"426b6f1e-f328-405c-86b7-00f2152e5567","Type":"ContainerDied","Data":"acb9994b690ade0839e35cb8c8789f17fd16418e9648eeae5d62734a23b3fdef"} Nov 01 04:23:39.923995 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.923976 6735 scope.go:122] "RemoveContainer" containerID="4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce" Nov 01 04:23:39.924355 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:39.924332 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce\": container with ID starting with 4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce not found: ID does not exist" containerID="4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce" Nov 01 04:23:39.924446 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.924366 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce"} err="failed to get container status \"4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce\": rpc error: code = NotFound desc = could not find container \"4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce\": container with ID starting with 4c76c32d0c392b5b045a830d6a0d994b662b396e5469fcdc704f1966eed483ce not found: ID does not exist" Nov 01 04:23:39.924446 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.924391 6735 scope.go:122] "RemoveContainer" containerID="89be19d9a55cf490e1f45509d2a1b83f9f94276e6d2de5ac6eee048b1b8b7e9d" Nov 01 04:23:39.934076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.934059 6735 scope.go:122] "RemoveContainer" containerID="c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc" Nov 01 04:23:39.942553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.942535 6735 scope.go:122] "RemoveContainer" containerID="c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc" Nov 01 04:23:39.942809 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:39.942791 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc\": container with ID starting with c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc not found: ID does not exist" containerID="c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc" Nov 01 04:23:39.942855 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:39.942815 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc"} err="failed to get container status \"c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc\": rpc error: code = NotFound desc = could not find container \"c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc\": container with ID starting with c496883af4631de385b87ea21d4b524011778c362b24b0a64871138934aa27dc not found: ID does not exist" Nov 01 04:23:40.067594 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.067507 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd"] Nov 01 04:23:40.071423 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.071398 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3a348f-8554b76b5c-llmsd"] Nov 01 04:23:40.174575 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.174539 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m"] Nov 01 04:23:40.178273 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.178244 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3a348f-d59b57b75-9sl2m"] Nov 01 04:23:40.281274 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.281239 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67"] Nov 01 04:23:40.284559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.284535 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3a348f-57dd9b44-dbq67"] Nov 01 04:23:40.526895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.526861 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426b6f1e-f328-405c-86b7-00f2152e5567" path="/var/lib/kubelet/pods/426b6f1e-f328-405c-86b7-00f2152e5567/volumes" Nov 01 04:23:40.527326 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.527310 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89975435-db64-486e-bd27-b603abc3a312" path="/var/lib/kubelet/pods/89975435-db64-486e-bd27-b603abc3a312/volumes" Nov 01 04:23:40.527657 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.527645 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacc0e05-a8f1-414c-b12a-deed8686bf07" path="/var/lib/kubelet/pods/dacc0e05-a8f1-414c-b12a-deed8686bf07/volumes" Nov 01 04:23:40.892920 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.892843 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd"] Nov 01 04:23:40.991394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:40.991363 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8"] Nov 01 04:23:45.893470 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:45.893429 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" podUID="5f561522-b6dc-43b3-bf9e-75112df1c820" containerName="payload-processing" containerID="cri-o://6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6" gracePeriod=25 Nov 01 04:23:45.991931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:45.991886 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" podUID="c9b2f327-5382-426f-a613-1b9a8d3e4c85" containerName="payload-pre-processing" containerID="cri-o://6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9" gracePeriod=25 Nov 01 04:23:46.180079 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.180026 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:23:46.284025 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.284001 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:23:46.284175 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.284129 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f561522-b6dc-43b3-bf9e-75112df1c820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f561522-b6dc-43b3-bf9e-75112df1c820-plugins-config-volume\") pod \"5f561522-b6dc-43b3-bf9e-75112df1c820\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " Nov 01 04:23:46.284175 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.284173 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f561522-b6dc-43b3-bf9e-75112df1c820-kube-api-access-gbzxk\" (UniqueName: \"kubernetes.io/projected/5f561522-b6dc-43b3-bf9e-75112df1c820-kube-api-access-gbzxk\") pod \"5f561522-b6dc-43b3-bf9e-75112df1c820\" (UID: \"5f561522-b6dc-43b3-bf9e-75112df1c820\") " Nov 01 04:23:46.284496 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.284466 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f561522-b6dc-43b3-bf9e-75112df1c820-plugins-config-volume" pod "5f561522-b6dc-43b3-bf9e-75112df1c820" (UID: "5f561522-b6dc-43b3-bf9e-75112df1c820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:46.284626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.284612 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f561522-b6dc-43b3-bf9e-75112df1c820-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:46.286291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.286272 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f561522-b6dc-43b3-bf9e-75112df1c820-kube-api-access-gbzxk" pod "5f561522-b6dc-43b3-bf9e-75112df1c820" (UID: "5f561522-b6dc-43b3-bf9e-75112df1c820"). InnerVolumeSpecName "kube-api-access-gbzxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:46.385792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.385757 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b2f327-5382-426f-a613-1b9a8d3e4c85-kube-api-access-6lckw\" (UniqueName: \"kubernetes.io/projected/c9b2f327-5382-426f-a613-1b9a8d3e4c85-kube-api-access-6lckw\") pod \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " Nov 01 04:23:46.385984 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.385862 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b2f327-5382-426f-a613-1b9a8d3e4c85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b2f327-5382-426f-a613-1b9a8d3e4c85-plugins-config-volume\") pod \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\" (UID: \"c9b2f327-5382-426f-a613-1b9a8d3e4c85\") " Nov 01 04:23:46.386242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.386223 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzxk\" (UniqueName: \"kubernetes.io/projected/5f561522-b6dc-43b3-bf9e-75112df1c820-kube-api-access-gbzxk\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:46.386350 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.386253 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b2f327-5382-426f-a613-1b9a8d3e4c85-plugins-config-volume" pod "c9b2f327-5382-426f-a613-1b9a8d3e4c85" (UID: "c9b2f327-5382-426f-a613-1b9a8d3e4c85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:46.388001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.387981 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b2f327-5382-426f-a613-1b9a8d3e4c85-kube-api-access-6lckw" pod "c9b2f327-5382-426f-a613-1b9a8d3e4c85" (UID: "c9b2f327-5382-426f-a613-1b9a8d3e4c85"). InnerVolumeSpecName "kube-api-access-6lckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:46.487657 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.487621 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b2f327-5382-426f-a613-1b9a8d3e4c85-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:46.487657 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.487651 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lckw\" (UniqueName: \"kubernetes.io/projected/c9b2f327-5382-426f-a613-1b9a8d3e4c85-kube-api-access-6lckw\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:46.967159 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.967111 6735 generic.go:356] "Generic (PLEG): container finished" podID="c9b2f327-5382-426f-a613-1b9a8d3e4c85" containerID="6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9" exitCode=0 Nov 01 04:23:46.967621 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.967330 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" Nov 01 04:23:46.967621 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.967484 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" event={"ID":"c9b2f327-5382-426f-a613-1b9a8d3e4c85","Type":"ContainerDied","Data":"6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9"} Nov 01 04:23:46.967621 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.967588 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8" event={"ID":"c9b2f327-5382-426f-a613-1b9a8d3e4c85","Type":"ContainerDied","Data":"a1e3d59f7d4c9403dc7f0453f36653a8cb254d6c073bf7c5c80880a046989d34"} Nov 01 04:23:46.967785 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.967632 6735 scope.go:122] "RemoveContainer" containerID="6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9" Nov 01 04:23:46.969708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.969484 6735 generic.go:356] "Generic (PLEG): container finished" podID="5f561522-b6dc-43b3-bf9e-75112df1c820" containerID="6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6" exitCode=0 Nov 01 04:23:46.969708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.969568 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" Nov 01 04:23:46.969708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.969664 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" event={"ID":"5f561522-b6dc-43b3-bf9e-75112df1c820","Type":"ContainerDied","Data":"6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6"} Nov 01 04:23:46.969708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.969705 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd" event={"ID":"5f561522-b6dc-43b3-bf9e-75112df1c820","Type":"ContainerDied","Data":"27892ac41273fae0f7fdabac6eab36f0d592dfaac5a57b1fa87ee1ea1f4747f4"} Nov 01 04:23:46.979112 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.979092 6735 scope.go:122] "RemoveContainer" containerID="6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9" Nov 01 04:23:46.979434 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:46.979403 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9\": container with ID starting with 6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9 not found: ID does not exist" containerID="6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9" Nov 01 04:23:46.979559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.979438 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9"} err="failed to get container status \"6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9\": rpc error: code = NotFound desc = could not find container \"6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9\": container with ID starting with 6197897ec6c12b367fb092897bf05c59ac9e28920f1c2dc1ecbf158e16d809e9 not found: ID does not exist" Nov 01 04:23:46.979559 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.979456 6735 scope.go:122] "RemoveContainer" containerID="6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6" Nov 01 04:23:46.989542 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.989524 6735 scope.go:122] "RemoveContainer" containerID="6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6" Nov 01 04:23:46.989856 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:23:46.989837 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6\": container with ID starting with 6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6 not found: ID does not exist" containerID="6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6" Nov 01 04:23:46.989911 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:46.989864 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6"} err="failed to get container status \"6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6\": rpc error: code = NotFound desc = could not find container \"6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6\": container with ID starting with 6ff26a1ed31567e0d5c6cca5908cf0b30e3235b75eeeaa6ce3df6f672fc4d4a6 not found: ID does not exist" Nov 01 04:23:47.122296 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:47.122262 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd"] Nov 01 04:23:47.168576 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:47.168483 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e7c9be-565c8b6885-2x6hd"] Nov 01 04:23:47.227499 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:47.227467 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8"] Nov 01 04:23:47.231378 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:47.231354 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e7c9be-6f96b9fbff-8w9d8"] Nov 01 04:23:48.525715 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:48.525680 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f561522-b6dc-43b3-bf9e-75112df1c820" path="/var/lib/kubelet/pods/5f561522-b6dc-43b3-bf9e-75112df1c820/volumes" Nov 01 04:23:48.526113 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:48.526098 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b2f327-5382-426f-a613-1b9a8d3e4c85" path="/var/lib/kubelet/pods/c9b2f327-5382-426f-a613-1b9a8d3e4c85/volumes" Nov 01 04:23:53.699890 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:53.699856 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:23:55.751685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:55.751649 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq"] Nov 01 04:23:55.752177 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:55.751841 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" podUID="9509d859-1099-41f5-8466-aad7e8d13f8d" containerName="maas-api" containerID="cri-o://300d3a1790e66ae6a778c0efa6fa5835deb249003a127dad7d7aec58b986d371" gracePeriod=30 Nov 01 04:23:56.033782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.033753 6735 generic.go:356] "Generic (PLEG): container finished" podID="9509d859-1099-41f5-8466-aad7e8d13f8d" containerID="300d3a1790e66ae6a778c0efa6fa5835deb249003a127dad7d7aec58b986d371" exitCode=0 Nov 01 04:23:56.033909 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.033802 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" event={"ID":"9509d859-1099-41f5-8466-aad7e8d13f8d","Type":"ContainerDied","Data":"300d3a1790e66ae6a778c0efa6fa5835deb249003a127dad7d7aec58b986d371"} Nov 01 04:23:56.033909 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.033841 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" event={"ID":"9509d859-1099-41f5-8466-aad7e8d13f8d","Type":"ContainerDied","Data":"818d057890ae96aa3ab1dcf668c5ff4d196d98cb765bdb58115bd8970dde6bd7"} Nov 01 04:23:56.033909 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.033856 6735 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="818d057890ae96aa3ab1dcf668c5ff4d196d98cb765bdb58115bd8970dde6bd7" Nov 01 04:23:56.042468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.042442 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:23:56.105673 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.105641 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls\") pod \"9509d859-1099-41f5-8466-aad7e8d13f8d\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " Nov 01 04:23:56.105882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.105776 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9509d859-1099-41f5-8466-aad7e8d13f8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9509d859-1099-41f5-8466-aad7e8d13f8d-openshift-service-ca\") pod \"9509d859-1099-41f5-8466-aad7e8d13f8d\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " Nov 01 04:23:56.105882 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.105867 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-metrics-certs\") pod \"9509d859-1099-41f5-8466-aad7e8d13f8d\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " Nov 01 04:23:56.105978 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.105909 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9509d859-1099-41f5-8466-aad7e8d13f8d-kube-api-access-8zmbr\" (UniqueName: \"kubernetes.io/projected/9509d859-1099-41f5-8466-aad7e8d13f8d-kube-api-access-8zmbr\") pod \"9509d859-1099-41f5-8466-aad7e8d13f8d\" (UID: \"9509d859-1099-41f5-8466-aad7e8d13f8d\") " Nov 01 04:23:56.106200 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.106169 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9509d859-1099-41f5-8466-aad7e8d13f8d-openshift-service-ca" pod "9509d859-1099-41f5-8466-aad7e8d13f8d" (UID: "9509d859-1099-41f5-8466-aad7e8d13f8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:23:56.106420 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.106398 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9509d859-1099-41f5-8466-aad7e8d13f8d-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:56.108070 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.108011 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls" pod "9509d859-1099-41f5-8466-aad7e8d13f8d" (UID: "9509d859-1099-41f5-8466-aad7e8d13f8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:23:56.108149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.108094 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-metrics-certs" pod "9509d859-1099-41f5-8466-aad7e8d13f8d" (UID: "9509d859-1099-41f5-8466-aad7e8d13f8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:23:56.108149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.108104 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9509d859-1099-41f5-8466-aad7e8d13f8d-kube-api-access-8zmbr" pod "9509d859-1099-41f5-8466-aad7e8d13f8d" (UID: "9509d859-1099-41f5-8466-aad7e8d13f8d"). InnerVolumeSpecName "kube-api-access-8zmbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:23:56.211931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.211844 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:56.211931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.211886 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9509d859-1099-41f5-8466-aad7e8d13f8d-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:56.211931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:56.211904 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmbr\" (UniqueName: \"kubernetes.io/projected/9509d859-1099-41f5-8466-aad7e8d13f8d-kube-api-access-8zmbr\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:23:57.039316 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:57.039274 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq" Nov 01 04:23:57.192926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:57.192891 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq"] Nov 01 04:23:57.239232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:57.239199 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a7b5f0-6bb458c9c6-7vjgq"] Nov 01 04:23:58.525984 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:23:58.525942 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9509d859-1099-41f5-8466-aad7e8d13f8d" path="/var/lib/kubelet/pods/9509d859-1099-41f5-8466-aad7e8d13f8d/volumes" Nov 01 04:24:02.496412 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.496375 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7"] Nov 01 04:24:02.496817 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.496621 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" podUID="ea5764f5-962b-4b4e-9558-272684c3dd5d" containerName="maas-api" containerID="cri-o://9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1" gracePeriod=30 Nov 01 04:24:02.806110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.806084 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:24:02.992497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.992466 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5764f5-962b-4b4e-9558-272684c3dd5d-kube-api-access-tjrmp\" (UniqueName: \"kubernetes.io/projected/ea5764f5-962b-4b4e-9558-272684c3dd5d-kube-api-access-tjrmp\") pod \"ea5764f5-962b-4b4e-9558-272684c3dd5d\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " Nov 01 04:24:02.992664 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.992573 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls\") pod \"ea5764f5-962b-4b4e-9558-272684c3dd5d\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " Nov 01 04:24:02.992664 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.992604 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5764f5-962b-4b4e-9558-272684c3dd5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5764f5-962b-4b4e-9558-272684c3dd5d-openshift-service-ca\") pod \"ea5764f5-962b-4b4e-9558-272684c3dd5d\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " Nov 01 04:24:02.992664 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.992639 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-metrics-certs\") pod \"ea5764f5-962b-4b4e-9558-272684c3dd5d\" (UID: \"ea5764f5-962b-4b4e-9558-272684c3dd5d\") " Nov 01 04:24:02.992941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.992918 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5764f5-962b-4b4e-9558-272684c3dd5d-openshift-service-ca" pod "ea5764f5-962b-4b4e-9558-272684c3dd5d" (UID: "ea5764f5-962b-4b4e-9558-272684c3dd5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:24:02.993154 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.993131 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5764f5-962b-4b4e-9558-272684c3dd5d-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:02.995287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.995234 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-metrics-certs" pod "ea5764f5-962b-4b4e-9558-272684c3dd5d" (UID: "ea5764f5-962b-4b4e-9558-272684c3dd5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:24:02.995484 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.995459 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5764f5-962b-4b4e-9558-272684c3dd5d-kube-api-access-tjrmp" pod "ea5764f5-962b-4b4e-9558-272684c3dd5d" (UID: "ea5764f5-962b-4b4e-9558-272684c3dd5d"). InnerVolumeSpecName "kube-api-access-tjrmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:24:02.995574 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:02.995484 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls" pod "ea5764f5-962b-4b4e-9558-272684c3dd5d" (UID: "ea5764f5-962b-4b4e-9558-272684c3dd5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:24:03.001727 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:24:03.001699 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:24:03.075388 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.075310 6735 generic.go:356] "Generic (PLEG): container finished" podID="ea5764f5-962b-4b4e-9558-272684c3dd5d" containerID="9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1" exitCode=0 Nov 01 04:24:03.075388 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.075362 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" Nov 01 04:24:03.075564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.075396 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" event={"ID":"ea5764f5-962b-4b4e-9558-272684c3dd5d","Type":"ContainerDied","Data":"9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1"} Nov 01 04:24:03.075564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.075450 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7" event={"ID":"ea5764f5-962b-4b4e-9558-272684c3dd5d","Type":"ContainerDied","Data":"15e982187b9f67a30f1b6e7e9bd154fdfa134810298f4ca7a86927bad03316ba"} Nov 01 04:24:03.075564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.075476 6735 scope.go:122] "RemoveContainer" containerID="9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1" Nov 01 04:24:03.085889 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.085869 6735 scope.go:122] "RemoveContainer" containerID="9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1" Nov 01 04:24:03.086186 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:24:03.086168 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1\": container with ID starting with 9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1 not found: ID does not exist" containerID="9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1" Nov 01 04:24:03.086259 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.086200 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1"} err="failed to get container status \"9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1\": rpc error: code = NotFound desc = could not find container \"9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1\": container with ID starting with 9353984ab429aeabf3a0c08b1cfa70816c971d55ff999991ad14a88ea88c70c1 not found: ID does not exist" Nov 01 04:24:03.093760 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.093740 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:03.093834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.093761 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea5764f5-962b-4b4e-9558-272684c3dd5d-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:03.093834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.093773 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrmp\" (UniqueName: \"kubernetes.io/projected/ea5764f5-962b-4b4e-9558-272684c3dd5d-kube-api-access-tjrmp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:03.232662 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.232627 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7"] Nov 01 04:24:03.236325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:03.236297 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ee0f8-5467c86547-6psl7"] Nov 01 04:24:04.526805 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:04.526769 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5764f5-962b-4b4e-9558-272684c3dd5d" path="/var/lib/kubelet/pods/ea5764f5-962b-4b4e-9558-272684c3dd5d/volumes" Nov 01 04:24:16.012304 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012269 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x"] Nov 01 04:24:16.012825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012809 6735 cpu_manager.go:414] "containerMap: removing container" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" Nov 01 04:24:16.012825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012825 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012838 6735 cpu_manager.go:414] "containerMap: removing container" podUID="b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" containerName="maas-api" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012843 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" containerName="maas-api" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012856 6735 cpu_manager.go:414] "containerMap: removing container" podUID="89975435-db64-486e-bd27-b603abc3a312" containerName="maas-api" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012861 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89975435-db64-486e-bd27-b603abc3a312" containerName="maas-api" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012867 6735 cpu_manager.go:414] "containerMap: removing container" podUID="9e3aec01-79aa-48a7-98be-34abc62658d6" containerName="payload-processing" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012872 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3aec01-79aa-48a7-98be-34abc62658d6" containerName="payload-processing" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012879 6735 cpu_manager.go:414] "containerMap: removing container" podUID="d52635e3-4499-422b-a147-e66dbcc499f9" containerName="payload-pre-processing" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012884 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52635e3-4499-422b-a147-e66dbcc499f9" containerName="payload-pre-processing" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012896 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012901 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012911 6735 cpu_manager.go:414] "containerMap: removing container" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012917 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" Nov 01 04:24:16.012922 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012927 6735 cpu_manager.go:414] "containerMap: removing container" podUID="ea5764f5-962b-4b4e-9558-272684c3dd5d" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012933 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5764f5-962b-4b4e-9558-272684c3dd5d" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012941 6735 cpu_manager.go:414] "containerMap: removing container" podUID="5f561522-b6dc-43b3-bf9e-75112df1c820" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012945 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f561522-b6dc-43b3-bf9e-75112df1c820" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012952 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c9c74d9c-c29c-4943-adaf-34dc271e5afc" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012956 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c74d9c-c29c-4943-adaf-34dc271e5afc" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012967 6735 cpu_manager.go:414] "containerMap: removing container" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012971 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012981 6735 cpu_manager.go:414] "containerMap: removing container" podUID="9509d859-1099-41f5-8466-aad7e8d13f8d" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012988 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9509d859-1099-41f5-8466-aad7e8d13f8d" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.012996 6735 cpu_manager.go:414] "containerMap: removing container" podUID="dacc0e05-a8f1-414c-b12a-deed8686bf07" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013001 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacc0e05-a8f1-414c-b12a-deed8686bf07" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013008 6735 cpu_manager.go:414] "containerMap: removing container" podUID="5b94e72f-f79b-4be1-bd80-b017b412f0bc" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013013 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b94e72f-f79b-4be1-bd80-b017b412f0bc" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013023 6735 cpu_manager.go:414] "containerMap: removing container" podUID="b4e528ab-13d2-4e11-bd51-bb2141f465bf" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013028 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e528ab-13d2-4e11-bd51-bb2141f465bf" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013048 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c9b2f327-5382-426f-a613-1b9a8d3e4c85" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013052 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b2f327-5382-426f-a613-1b9a8d3e4c85" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013062 6735 cpu_manager.go:414] "containerMap: removing container" podUID="426b6f1e-f328-405c-86b7-00f2152e5567" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013069 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426b6f1e-f328-405c-86b7-00f2152e5567" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013157 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013167 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5764f5-962b-4b4e-9558-272684c3dd5d" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013174 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f561522-b6dc-43b3-bf9e-75112df1c820" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013180 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c74d9c-c29c-4943-adaf-34dc271e5afc" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013188 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="9509d859-1099-41f5-8466-aad7e8d13f8d" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013193 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacc0e05-a8f1-414c-b12a-deed8686bf07" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013200 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b94e72f-f79b-4be1-bd80-b017b412f0bc" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013207 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e528ab-13d2-4e11-bd51-bb2141f465bf" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013213 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b2f327-5382-426f-a613-1b9a8d3e4c85" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013219 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="426b6f1e-f328-405c-86b7-00f2152e5567" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013226 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf893cd-f7ae-43fc-b66f-0840ad6cc48e" containerName="istio-proxy" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013232 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ab6e5f-8227-44f6-84e3-921ef42ffe3e" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013238 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="89975435-db64-486e-bd27-b603abc3a312" containerName="maas-api" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013243 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3aec01-79aa-48a7-98be-34abc62658d6" containerName="payload-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013249 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52635e3-4499-422b-a147-e66dbcc499f9" containerName="payload-pre-processing" Nov 01 04:24:16.013425 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.013255 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a15781-c068-4965-87da-93bbaa57cc01" containerName="istio-proxy" Nov 01 04:24:16.018480 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.018462 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.085000 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.084943 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmw9m\" (UniqueName: \"kubernetes.io/projected/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-kube-api-access-wmw9m\") pod \"payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.085231 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.085108 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.115837 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.115800 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-550b8a-dockercfg-m8lss\"" Nov 01 04:24:16.123504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.119734 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-550b8a\"" Nov 01 04:24:16.123504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.121791 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x"] Nov 01 04:24:16.172185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.172150 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt"] Nov 01 04:24:16.172818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.172801 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc8b0b4-6032-451e-9cc8-0e666197d896" containerName="istio-proxy" Nov 01 04:24:16.177122 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.177083 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.186293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.186264 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.186447 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.186431 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmw9m\" (UniqueName: \"kubernetes.io/projected/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-kube-api-access-wmw9m\") pod \"payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.187008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.186988 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.228865 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.228831 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt"] Nov 01 04:24:16.244176 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.244149 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmw9m\" (UniqueName: \"kubernetes.io/projected/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-kube-api-access-wmw9m\") pod \"payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.287919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.287839 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4778da-864c-47de-b518-a01e669b5b45-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.287919 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.287881 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pp2l\" (UniqueName: \"kubernetes.io/projected/fe4778da-864c-47de-b518-a01e669b5b45-kube-api-access-9pp2l\") pod \"payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.331881 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.331851 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:16.389012 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.388977 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4778da-864c-47de-b518-a01e669b5b45-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.389184 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.389020 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pp2l\" (UniqueName: \"kubernetes.io/projected/fe4778da-864c-47de-b518-a01e669b5b45-kube-api-access-9pp2l\") pod \"payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.389985 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.389942 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4778da-864c-47de-b518-a01e669b5b45-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.446861 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.446835 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pp2l\" (UniqueName: \"kubernetes.io/projected/fe4778da-864c-47de-b518-a01e669b5b45-kube-api-access-9pp2l\") pod \"payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.492668 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.492634 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:16.702478 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.702444 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x"] Nov 01 04:24:16.704503 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:24:16.704473 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a45e4da_0cd9_4fed_b37e_aff6ea285f91.slice/crio-e7815cb649923d80248169b03e1469627c1141ddaeea8fd2bc8d2c0c05ae2347 WatchSource:0}: Error finding container e7815cb649923d80248169b03e1469627c1141ddaeea8fd2bc8d2c0c05ae2347: Status 404 returned error can't find the container with id e7815cb649923d80248169b03e1469627c1141ddaeea8fd2bc8d2c0c05ae2347 Nov 01 04:24:16.865856 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:16.865791 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt"] Nov 01 04:24:16.868561 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:24:16.868518 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4778da_864c_47de_b518_a01e669b5b45.slice/crio-fb3f8caadfbdb4567a4486cf99b26d1e7ff71334df7c864fad3a24a24ecaab12 WatchSource:0}: Error finding container fb3f8caadfbdb4567a4486cf99b26d1e7ff71334df7c864fad3a24a24ecaab12: Status 404 returned error can't find the container with id fb3f8caadfbdb4567a4486cf99b26d1e7ff71334df7c864fad3a24a24ecaab12 Nov 01 04:24:17.176325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.176234 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" event={"ID":"1a45e4da-0cd9-4fed-b37e-aff6ea285f91","Type":"ContainerStarted","Data":"9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a"} Nov 01 04:24:17.176325 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.176299 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" event={"ID":"1a45e4da-0cd9-4fed-b37e-aff6ea285f91","Type":"ContainerStarted","Data":"e7815cb649923d80248169b03e1469627c1141ddaeea8fd2bc8d2c0c05ae2347"} Nov 01 04:24:17.176809 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.176413 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:17.178262 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.178239 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" event={"ID":"fe4778da-864c-47de-b518-a01e669b5b45","Type":"ContainerStarted","Data":"8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891"} Nov 01 04:24:17.178371 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.178277 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" event={"ID":"fe4778da-864c-47de-b518-a01e669b5b45","Type":"ContainerStarted","Data":"fb3f8caadfbdb4567a4486cf99b26d1e7ff71334df7c864fad3a24a24ecaab12"} Nov 01 04:24:17.178371 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.178297 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:17.285968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.285915 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" podStartSLOduration=2.285901197 podStartE2EDuration="2.285901197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:24:17.284234969 +0000 UTC m=+2468.873329598" watchObservedRunningTime="2026-11-01 04:24:17.285901197 +0000 UTC m=+2468.874995825" Nov 01 04:24:17.393880 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:17.393823 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" podStartSLOduration=1.39380533 podStartE2EDuration="1.39380533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:24:17.392720871 +0000 UTC m=+2468.981815500" watchObservedRunningTime="2026-11-01 04:24:17.39380533 +0000 UTC m=+2468.982899960" Nov 01 04:24:23.884528 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:23.884493 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p"] Nov 01 04:24:23.888210 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:23.888193 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:23.937164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:23.937138 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ad9e22\"" Nov 01 04:24:24.000094 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:23.995556 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ad9e22-dockercfg-b2rtt\"" Nov 01 04:24:24.009549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.009520 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p"] Nov 01 04:24:24.010492 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.010465 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ecd83a-560d-4e93-81cb-3474b8d9881b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:24.010824 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.010799 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgr2\" (UniqueName: \"kubernetes.io/projected/01ecd83a-560d-4e93-81cb-3474b8d9881b-kube-api-access-zlgr2\") pod \"payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:24.055555 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.055522 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587"] Nov 01 04:24:24.059815 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.059793 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:24.112579 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.112549 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ecd83a-560d-4e93-81cb-3474b8d9881b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:24.112759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.112599 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgr2\" (UniqueName: \"kubernetes.io/projected/01ecd83a-560d-4e93-81cb-3474b8d9881b-kube-api-access-zlgr2\") pod \"payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:24.113409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.113377 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ecd83a-560d-4e93-81cb-3474b8d9881b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:24.114513 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.114490 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587"] Nov 01 04:24:24.170680 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.170603 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgr2\" (UniqueName: \"kubernetes.io/projected/01ecd83a-560d-4e93-81cb-3474b8d9881b-kube-api-access-zlgr2\") pod \"payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:24.202500 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.202472 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:24.213590 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.213564 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aa7198-e7c4-43f4-a101-f80f82bf3f70-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:24.213685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.213624 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwtq\" (UniqueName: \"kubernetes.io/projected/97aa7198-e7c4-43f4-a101-f80f82bf3f70-kube-api-access-pcwtq\") pod \"payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:24.314419 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.314377 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aa7198-e7c4-43f4-a101-f80f82bf3f70-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:24.314419 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.314426 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwtq\" (UniqueName: \"kubernetes.io/projected/97aa7198-e7c4-43f4-a101-f80f82bf3f70-kube-api-access-pcwtq\") pod \"payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:24.315146 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.315121 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aa7198-e7c4-43f4-a101-f80f82bf3f70-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:24.374813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.374784 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwtq\" (UniqueName: \"kubernetes.io/projected/97aa7198-e7c4-43f4-a101-f80f82bf3f70-kube-api-access-pcwtq\") pod \"payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:24.570508 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.570479 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p"] Nov 01 04:24:24.572217 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:24:24.572189 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ecd83a_560d_4e93_81cb_3474b8d9881b.slice/crio-7b53e815a4e401d86987e8020d4824c2161dfbd7797133b09a5691ec8812e58b WatchSource:0}: Error finding container 7b53e815a4e401d86987e8020d4824c2161dfbd7797133b09a5691ec8812e58b: Status 404 returned error can't find the container with id 7b53e815a4e401d86987e8020d4824c2161dfbd7797133b09a5691ec8812e58b Nov 01 04:24:24.674086 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:24.674059 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:25.042180 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.042145 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587"] Nov 01 04:24:25.044197 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:24:25.044143 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aa7198_e7c4_43f4_a101_f80f82bf3f70.slice/crio-ca85e5b0a5f90bffe3b40c596c0ba07a78ae3bddd894b1fa75fe6b5cba52fa08 WatchSource:0}: Error finding container ca85e5b0a5f90bffe3b40c596c0ba07a78ae3bddd894b1fa75fe6b5cba52fa08: Status 404 returned error can't find the container with id ca85e5b0a5f90bffe3b40c596c0ba07a78ae3bddd894b1fa75fe6b5cba52fa08 Nov 01 04:24:25.231182 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.231147 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" event={"ID":"01ecd83a-560d-4e93-81cb-3474b8d9881b","Type":"ContainerStarted","Data":"0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797"} Nov 01 04:24:25.231375 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.231208 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" event={"ID":"01ecd83a-560d-4e93-81cb-3474b8d9881b","Type":"ContainerStarted","Data":"7b53e815a4e401d86987e8020d4824c2161dfbd7797133b09a5691ec8812e58b"} Nov 01 04:24:25.231375 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.231336 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:25.232755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.232728 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" event={"ID":"97aa7198-e7c4-43f4-a101-f80f82bf3f70","Type":"ContainerStarted","Data":"b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb"} Nov 01 04:24:25.232876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.232779 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" event={"ID":"97aa7198-e7c4-43f4-a101-f80f82bf3f70","Type":"ContainerStarted","Data":"ca85e5b0a5f90bffe3b40c596c0ba07a78ae3bddd894b1fa75fe6b5cba52fa08"} Nov 01 04:24:25.232876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.232809 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:25.341474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.341373 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" podStartSLOduration=2.341359711 podStartE2EDuration="2.341359711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:24:25.340085222 +0000 UTC m=+2476.929179848" watchObservedRunningTime="2026-11-01 04:24:25.341359711 +0000 UTC m=+2476.930454339" Nov 01 04:24:25.456460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:25.453953 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" podStartSLOduration=1.453931532 podStartE2EDuration="1.453931532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:24:25.448211004 +0000 UTC m=+2477.037305632" watchObservedRunningTime="2026-11-01 04:24:25.453931532 +0000 UTC m=+2477.043026165" Nov 01 04:24:28.190276 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:28.190244 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:24:28.193276 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:28.193248 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:24:33.001619 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:24:33.001574 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:24:36.241441 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:36.241411 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:24:36.242000 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:36.241600 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:24:36.454578 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:36.454528 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96"] Nov 01 04:24:36.454948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:36.454811 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" containerID="cri-o://8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c" gracePeriod=30 Nov 01 04:24:41.720564 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.720505 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:24:41.822232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822204 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-envoy\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822419 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822272 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-credential-socket\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822419 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822352 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-token\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-token\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822419 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822384 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-data\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822419 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822411 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-certs\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822432 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istiod-ca-cert\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822495 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-socket\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822531 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-podinfo\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822566 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-kube-api-access-287mc\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-kube-api-access-287mc\") pod \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\" (UID: \"1adf4f20-5ee4-4dad-9b7f-e68f4007c068\") " Nov 01 04:24:41.822665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822604 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-credential-socket" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:24:41.823109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822708 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-certs" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:24:41.823109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822721 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-data" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:24:41.823109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.822918 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istiod-ca-cert" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:24:41.823109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.823085 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.823109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.823105 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.823387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.823115 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.823387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.823123 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.823501 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.823398 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-socket" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:24:41.824684 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.824657 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-envoy" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:24:41.824810 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.824740 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-podinfo" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:24:41.824810 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.824794 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-token" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:24:41.825001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.824983 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-kube-api-access-287mc" pod "1adf4f20-5ee4-4dad-9b7f-e68f4007c068" (UID: "1adf4f20-5ee4-4dad-9b7f-e68f4007c068"). InnerVolumeSpecName "kube-api-access-287mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:24:41.931366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.925191 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.931366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.925228 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.931366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.925245 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-287mc\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-kube-api-access-287mc\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.931366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.925261 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:41.931366 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:41.925281 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adf4f20-5ee4-4dad-9b7f-e68f4007c068-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:42.328158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.328120 6735 generic.go:356] "Generic (PLEG): container finished" podID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerID="8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c" exitCode=0 Nov 01 04:24:42.328374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.328192 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" Nov 01 04:24:42.328374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.328205 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" event={"ID":"1adf4f20-5ee4-4dad-9b7f-e68f4007c068","Type":"ContainerDied","Data":"8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c"} Nov 01 04:24:42.328374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.328306 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96" event={"ID":"1adf4f20-5ee4-4dad-9b7f-e68f4007c068","Type":"ContainerDied","Data":"a738ffc37d23d35bd0ef0e47e87a392c1a3dc66d304d76aa6e0541a83958d730"} Nov 01 04:24:42.328374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.328338 6735 scope.go:122] "RemoveContainer" containerID="8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c" Nov 01 04:24:42.338623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.338606 6735 scope.go:122] "RemoveContainer" containerID="7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778" Nov 01 04:24:42.347525 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.347506 6735 scope.go:122] "RemoveContainer" containerID="8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c" Nov 01 04:24:42.347756 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:24:42.347737 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c\": container with ID starting with 8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c not found: ID does not exist" containerID="8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c" Nov 01 04:24:42.347808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.347764 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c"} err="failed to get container status \"8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c\": rpc error: code = NotFound desc = could not find container \"8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c\": container with ID starting with 8f4b254241be0ea911fab3fe09c190baaa35c435795444a42e0131bdd3f1cf1c not found: ID does not exist" Nov 01 04:24:42.347808 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.347779 6735 scope.go:122] "RemoveContainer" containerID="7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778" Nov 01 04:24:42.347981 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:24:42.347959 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778\": container with ID starting with 7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778 not found: ID does not exist" containerID="7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778" Nov 01 04:24:42.348021 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.347987 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778"} err="failed to get container status \"7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778\": rpc error: code = NotFound desc = could not find container \"7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778\": container with ID starting with 7a65c60d4ac2bdd71fb41eb28bf2ef720149144aab9e623757cd39b1fb6fc778 not found: ID does not exist" Nov 01 04:24:42.542335 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.542300 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96"] Nov 01 04:24:42.587781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.587697 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e7c9be-openshift-default-6bbdf499f7-rkh96"] Nov 01 04:24:42.835173 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:42.835138 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:24:44.527288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:44.527251 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" path="/var/lib/kubelet/pods/1adf4f20-5ee4-4dad-9b7f-e68f4007c068/volumes" Nov 01 04:24:48.797589 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:48.797556 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk"] Nov 01 04:24:53.798506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:53.798442 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" podUID="bd44cb06-9266-4b28-a59c-91294da97f1f" containerName="payload-processing" containerID="cri-o://2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78" gracePeriod=25 Nov 01 04:24:54.059111 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.059056 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:24:54.166639 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.166609 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd44cb06-9266-4b28-a59c-91294da97f1f-kube-api-access-5m8l7\" (UniqueName: \"kubernetes.io/projected/bd44cb06-9266-4b28-a59c-91294da97f1f-kube-api-access-5m8l7\") pod \"bd44cb06-9266-4b28-a59c-91294da97f1f\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " Nov 01 04:24:54.166818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.166697 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd44cb06-9266-4b28-a59c-91294da97f1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd44cb06-9266-4b28-a59c-91294da97f1f-plugins-config-volume\") pod \"bd44cb06-9266-4b28-a59c-91294da97f1f\" (UID: \"bd44cb06-9266-4b28-a59c-91294da97f1f\") " Nov 01 04:24:54.167108 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.167062 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd44cb06-9266-4b28-a59c-91294da97f1f-plugins-config-volume" pod "bd44cb06-9266-4b28-a59c-91294da97f1f" (UID: "bd44cb06-9266-4b28-a59c-91294da97f1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:24:54.169352 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.169321 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd44cb06-9266-4b28-a59c-91294da97f1f-kube-api-access-5m8l7" pod "bd44cb06-9266-4b28-a59c-91294da97f1f" (UID: "bd44cb06-9266-4b28-a59c-91294da97f1f"). InnerVolumeSpecName "kube-api-access-5m8l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:24:54.268023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.267992 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd44cb06-9266-4b28-a59c-91294da97f1f-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:54.268023 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.268022 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m8l7\" (UniqueName: \"kubernetes.io/projected/bd44cb06-9266-4b28-a59c-91294da97f1f-kube-api-access-5m8l7\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:54.400510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.400423 6735 generic.go:356] "Generic (PLEG): container finished" podID="bd44cb06-9266-4b28-a59c-91294da97f1f" containerID="2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78" exitCode=0 Nov 01 04:24:54.400510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.400474 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" Nov 01 04:24:54.400510 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.400499 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" event={"ID":"bd44cb06-9266-4b28-a59c-91294da97f1f","Type":"ContainerDied","Data":"2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78"} Nov 01 04:24:54.400757 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.400550 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk" event={"ID":"bd44cb06-9266-4b28-a59c-91294da97f1f","Type":"ContainerDied","Data":"d528447032db1b8fcba943beea23945db63e6c13b20cc65bdb071911b059fcc5"} Nov 01 04:24:54.400757 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.400570 6735 scope.go:122] "RemoveContainer" containerID="2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78" Nov 01 04:24:54.410495 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.410480 6735 scope.go:122] "RemoveContainer" containerID="2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78" Nov 01 04:24:54.410766 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:24:54.410746 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78\": container with ID starting with 2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78 not found: ID does not exist" containerID="2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78" Nov 01 04:24:54.410811 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.410774 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78"} err="failed to get container status \"2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78\": rpc error: code = NotFound desc = could not find container \"2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78\": container with ID starting with 2cf04d9bf11bc7169ca102ca5da737e06b29f0247e67940e92196c913b19ef78 not found: ID does not exist" Nov 01 04:24:54.554604 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.554573 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk"] Nov 01 04:24:54.600498 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.600470 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-862ad1-6ff648ccc-6kcvk"] Nov 01 04:24:54.859659 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.859625 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76"] Nov 01 04:24:54.860087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.859838 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" podUID="1ca55b28-a359-47b3-bcb3-a355eea4d24a" containerName="maas-api" containerID="cri-o://f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474" gracePeriod=30 Nov 01 04:24:54.880228 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:54.880191 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" podUID="1ca55b28-a359-47b3-bcb3-a355eea4d24a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": dial tcp 10.134.0.146:8443: connect: connection refused" Nov 01 04:24:55.134141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.134119 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:24:55.186481 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.186454 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca55b28-a359-47b3-bcb3-a355eea4d24a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca55b28-a359-47b3-bcb3-a355eea4d24a-openshift-service-ca\") pod \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " Nov 01 04:24:55.186651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.186507 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls\") pod \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " Nov 01 04:24:55.186651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.186580 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-metrics-certs\") pod \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " Nov 01 04:24:55.186651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.186600 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca55b28-a359-47b3-bcb3-a355eea4d24a-kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/1ca55b28-a359-47b3-bcb3-a355eea4d24a-kube-api-access-zfmjm\") pod \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\" (UID: \"1ca55b28-a359-47b3-bcb3-a355eea4d24a\") " Nov 01 04:24:55.186918 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.186877 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca55b28-a359-47b3-bcb3-a355eea4d24a-openshift-service-ca" pod "1ca55b28-a359-47b3-bcb3-a355eea4d24a" (UID: "1ca55b28-a359-47b3-bcb3-a355eea4d24a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:24:55.188854 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.188829 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls" pod "1ca55b28-a359-47b3-bcb3-a355eea4d24a" (UID: "1ca55b28-a359-47b3-bcb3-a355eea4d24a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:24:55.188944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.188834 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca55b28-a359-47b3-bcb3-a355eea4d24a-kube-api-access-zfmjm" pod "1ca55b28-a359-47b3-bcb3-a355eea4d24a" (UID: "1ca55b28-a359-47b3-bcb3-a355eea4d24a"). InnerVolumeSpecName "kube-api-access-zfmjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:24:55.188944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.188868 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-metrics-certs" pod "1ca55b28-a359-47b3-bcb3-a355eea4d24a" (UID: "1ca55b28-a359-47b3-bcb3-a355eea4d24a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:24:55.288817 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.288772 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ca55b28-a359-47b3-bcb3-a355eea4d24a-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:55.288817 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.288812 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:55.289114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.288826 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ca55b28-a359-47b3-bcb3-a355eea4d24a-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:55.289114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.288842 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfmjm\" (UniqueName: \"kubernetes.io/projected/1ca55b28-a359-47b3-bcb3-a355eea4d24a-kube-api-access-zfmjm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:24:55.408199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.408110 6735 generic.go:356] "Generic (PLEG): container finished" podID="1ca55b28-a359-47b3-bcb3-a355eea4d24a" containerID="f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474" exitCode=0 Nov 01 04:24:55.408199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.408159 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" Nov 01 04:24:55.408199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.408187 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" event={"ID":"1ca55b28-a359-47b3-bcb3-a355eea4d24a","Type":"ContainerDied","Data":"f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474"} Nov 01 04:24:55.408529 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.408242 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76" event={"ID":"1ca55b28-a359-47b3-bcb3-a355eea4d24a","Type":"ContainerDied","Data":"8633e83b65ca0765c4e26d8dabec54a8cd08c3c62b83c74a5a8ec2abca6f5b68"} Nov 01 04:24:55.408529 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.408268 6735 scope.go:122] "RemoveContainer" containerID="f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474" Nov 01 04:24:55.418475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.418459 6735 scope.go:122] "RemoveContainer" containerID="f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474" Nov 01 04:24:55.418750 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:24:55.418731 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474\": container with ID starting with f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474 not found: ID does not exist" containerID="f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474" Nov 01 04:24:55.418797 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.418759 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474"} err="failed to get container status \"f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474\": rpc error: code = NotFound desc = could not find container \"f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474\": container with ID starting with f9023ebc4e84be1f4ae10c06a12deeb5cf87f603d93ab68698d336f072b70474 not found: ID does not exist" Nov 01 04:24:55.563273 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.563240 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76"] Nov 01 04:24:55.568877 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:55.568852 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0882f4-dd896684c-7bh76"] Nov 01 04:24:56.527295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:56.527264 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca55b28-a359-47b3-bcb3-a355eea4d24a" path="/var/lib/kubelet/pods/1ca55b28-a359-47b3-bcb3-a355eea4d24a/volumes" Nov 01 04:24:56.527765 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:24:56.527661 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd44cb06-9266-4b28-a59c-91294da97f1f" path="/var/lib/kubelet/pods/bd44cb06-9266-4b28-a59c-91294da97f1f/volumes" Nov 01 04:25:03.000817 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:25:03.000777 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:25:06.781843 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.781803 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj"] Nov 01 04:25:06.782582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782556 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" Nov 01 04:25:06.782582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782577 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" Nov 01 04:25:06.782738 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782598 6735 cpu_manager.go:414] "containerMap: removing container" podUID="bd44cb06-9266-4b28-a59c-91294da97f1f" containerName="payload-processing" Nov 01 04:25:06.782738 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782605 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd44cb06-9266-4b28-a59c-91294da97f1f" containerName="payload-processing" Nov 01 04:25:06.782738 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782630 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" Nov 01 04:25:06.782738 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782639 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" Nov 01 04:25:06.782738 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782648 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1ca55b28-a359-47b3-bcb3-a355eea4d24a" containerName="maas-api" Nov 01 04:25:06.782738 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782655 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca55b28-a359-47b3-bcb3-a355eea4d24a" containerName="maas-api" Nov 01 04:25:06.782930 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782788 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" Nov 01 04:25:06.782930 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782799 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd44cb06-9266-4b28-a59c-91294da97f1f" containerName="payload-processing" Nov 01 04:25:06.782930 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782810 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1adf4f20-5ee4-4dad-9b7f-e68f4007c068" containerName="istio-proxy" Nov 01 04:25:06.782930 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.782818 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca55b28-a359-47b3-bcb3-a355eea4d24a" containerName="maas-api" Nov 01 04:25:06.786589 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.786572 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:06.838830 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.838802 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-550b8a-serving-cert\"" Nov 01 04:25:06.841847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.841820 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj"] Nov 01 04:25:06.929167 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.929110 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-metrics-certs\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:06.929398 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.929354 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e600c-65ea-412b-b429-1da4c5ccc826-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:06.929516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.929406 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:06.929578 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:06.929527 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpm2\" (UniqueName: \"kubernetes.io/projected/435e600c-65ea-412b-b429-1da4c5ccc826-kube-api-access-qxpm2\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.031077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.031014 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e600c-65ea-412b-b429-1da4c5ccc826-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.031285 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.031077 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.031285 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.031133 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpm2\" (UniqueName: \"kubernetes.io/projected/435e600c-65ea-412b-b429-1da4c5ccc826-kube-api-access-qxpm2\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.031285 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:25:07.031198 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-serving-cert: secret "maas-api-e2e-praxis-a-550b8a-serving-cert" not found Nov 01 04:25:07.031285 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.031255 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-metrics-certs\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.031285 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:25:07.031283 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls podName:435e600c-65ea-412b-b429-1da4c5ccc826 nodeName:}" failed. No retries permitted until 2026-11-01 04:25:07.531261999 +0000 UTC m=+2519.120356625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls") pod "maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" (UID: "435e600c-65ea-412b-b429-1da4c5ccc826") : secret "maas-api-e2e-praxis-a-550b8a-serving-cert" not found Nov 01 04:25:07.031809 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.031789 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e600c-65ea-412b-b429-1da4c5ccc826-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.033794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.033743 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-metrics-certs\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.091309 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.091269 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpm2\" (UniqueName: \"kubernetes.io/projected/435e600c-65ea-412b-b429-1da4c5ccc826-kube-api-access-qxpm2\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.537736 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.537704 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.540317 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.540296 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls\") pod \"maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:07.700843 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:07.700809 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:08.070128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:08.070098 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj"] Nov 01 04:25:08.072189 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:25:08.072157 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435e600c_65ea_412b_b429_1da4c5ccc826.slice/crio-9f061fdcd21d6d794eb85f6ae7bb637234e9869b1d7c5729d97ce997c6a0175f WatchSource:0}: Error finding container 9f061fdcd21d6d794eb85f6ae7bb637234e9869b1d7c5729d97ce997c6a0175f: Status 404 returned error can't find the container with id 9f061fdcd21d6d794eb85f6ae7bb637234e9869b1d7c5729d97ce997c6a0175f Nov 01 04:25:08.073688 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:08.073661 6735 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 04:25:08.488706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:08.488670 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" event={"ID":"435e600c-65ea-412b-b429-1da4c5ccc826","Type":"ContainerStarted","Data":"9f061fdcd21d6d794eb85f6ae7bb637234e9869b1d7c5729d97ce997c6a0175f"} Nov 01 04:25:09.496721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:09.496681 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" event={"ID":"435e600c-65ea-412b-b429-1da4c5ccc826","Type":"ContainerStarted","Data":"fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422"} Nov 01 04:25:09.497207 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:09.496832 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:09.608953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:09.608901 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" podStartSLOduration=2.570819693 podStartE2EDuration="3.608888295s" totalImagesPullingTime="1.038068602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:25:09.606835238 +0000 UTC m=+2521.195929865" watchObservedRunningTime="2026-11-01 04:25:09.608888295 +0000 UTC m=+2521.197982952" Nov 01 04:25:09.984696 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:09.984661 6735 scope.go:122] "RemoveContainer" containerID="300d3a1790e66ae6a778c0efa6fa5835deb249003a127dad7d7aec58b986d371" Nov 01 04:25:15.509131 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.509021 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:25:15.606409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.606374 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt"] Nov 01 04:25:15.611604 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.611588 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.656452 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.656416 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt"] Nov 01 04:25:15.670281 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.670241 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1cdd1d\"" Nov 01 04:25:15.670281 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.670257 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1cdd1d-dockercfg-d9fgx\"" Nov 01 04:25:15.734248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.734195 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f724760-4640-41a2-b471-fe60addef235-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.734416 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.734352 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmd2d\" (UniqueName: \"kubernetes.io/projected/2f724760-4640-41a2-b471-fe60addef235-kube-api-access-qmd2d\") pod \"payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.753494 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.753368 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj"] Nov 01 04:25:15.772471 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.772394 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:15.818104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.818065 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj"] Nov 01 04:25:15.835599 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.835560 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f724760-4640-41a2-b471-fe60addef235-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.835766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.835671 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmd2d\" (UniqueName: \"kubernetes.io/projected/2f724760-4640-41a2-b471-fe60addef235-kube-api-access-qmd2d\") pod \"payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.836265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.836244 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f724760-4640-41a2-b471-fe60addef235-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.893696 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.893664 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmd2d\" (UniqueName: \"kubernetes.io/projected/2f724760-4640-41a2-b471-fe60addef235-kube-api-access-qmd2d\") pod \"payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.925523 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.925494 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:15.937378 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.937299 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxtg\" (UniqueName: \"kubernetes.io/projected/03003767-c63c-42f5-84b1-8956917491b0-kube-api-access-bsxtg\") pod \"payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:15.937378 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:15.937358 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03003767-c63c-42f5-84b1-8956917491b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:16.039137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.039028 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxtg\" (UniqueName: \"kubernetes.io/projected/03003767-c63c-42f5-84b1-8956917491b0-kube-api-access-bsxtg\") pod \"payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:16.039137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.039117 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03003767-c63c-42f5-84b1-8956917491b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:16.039782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.039763 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03003767-c63c-42f5-84b1-8956917491b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:16.097161 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.097112 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxtg\" (UniqueName: \"kubernetes.io/projected/03003767-c63c-42f5-84b1-8956917491b0-kube-api-access-bsxtg\") pod \"payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:16.292473 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.292442 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt"] Nov 01 04:25:16.294348 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:25:16.294323 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f724760_4640_41a2_b471_fe60addef235.slice/crio-453beca7de3d900bc9bb1fb2f086478a2586e93a803126864e8121644bbd9152 WatchSource:0}: Error finding container 453beca7de3d900bc9bb1fb2f086478a2586e93a803126864e8121644bbd9152: Status 404 returned error can't find the container with id 453beca7de3d900bc9bb1fb2f086478a2586e93a803126864e8121644bbd9152 Nov 01 04:25:16.393393 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.393368 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:16.537608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.537574 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" event={"ID":"2f724760-4640-41a2-b471-fe60addef235","Type":"ContainerStarted","Data":"5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73"} Nov 01 04:25:16.538020 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.537715 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:16.538020 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.537735 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" event={"ID":"2f724760-4640-41a2-b471-fe60addef235","Type":"ContainerStarted","Data":"453beca7de3d900bc9bb1fb2f086478a2586e93a803126864e8121644bbd9152"} Nov 01 04:25:16.646928 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.646822 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" podStartSLOduration=1.646808879 podStartE2EDuration="1.646808879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:25:16.646690665 +0000 UTC m=+2528.235785318" watchObservedRunningTime="2026-11-01 04:25:16.646808879 +0000 UTC m=+2528.235903544" Nov 01 04:25:16.775645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:16.775617 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj"] Nov 01 04:25:17.544675 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:17.544638 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" event={"ID":"03003767-c63c-42f5-84b1-8956917491b0","Type":"ContainerStarted","Data":"d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1"} Nov 01 04:25:17.545161 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:17.544698 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" event={"ID":"03003767-c63c-42f5-84b1-8956917491b0","Type":"ContainerStarted","Data":"5be72f031be4f331ca9e59dd2a7c7d852dc0f5c21d1cb88ece995ad3eb7e57b0"} Nov 01 04:25:18.441900 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.441845 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" podStartSLOduration=3.441832008 podStartE2EDuration="3.441832008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:25:17.656232729 +0000 UTC m=+2529.245327357" watchObservedRunningTime="2026-11-01 04:25:18.441832008 +0000 UTC m=+2530.030926637" Nov 01 04:25:18.442457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.442442 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:25:18.559202 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.559167 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:18.697864 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.697775 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk"] Nov 01 04:25:18.701788 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.701767 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.752600 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.752564 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ad9e22-serving-cert\"" Nov 01 04:25:18.759349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.758217 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk"] Nov 01 04:25:18.886909 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.886876 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj426\" (UniqueName: \"kubernetes.io/projected/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-kube-api-access-jj426\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.887089 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.886926 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-metrics-certs\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.887142 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.887072 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.887188 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.887148 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.988477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.988440 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-metrics-certs\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.988651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.988507 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.988651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.988536 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.988651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.988634 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj426\" (UniqueName: \"kubernetes.io/projected/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-kube-api-access-jj426\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.988836 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:25:18.988731 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-serving-cert: secret "maas-api-e2e-praxis-a-ad9e22-serving-cert" not found Nov 01 04:25:18.988836 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:25:18.988814 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls podName:cc0d6f23-c9f0-4c32-9472-4553b5cf45e6 nodeName:}" failed. No retries permitted until 2026-11-01 04:25:19.48879216 +0000 UTC m=+2531.077886782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls") pod "maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" (UID: "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6") : secret "maas-api-e2e-praxis-a-ad9e22-serving-cert" not found Nov 01 04:25:18.989299 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.989276 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:18.991725 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:18.991698 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-metrics-certs\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:19.046950 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:19.046913 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj426\" (UniqueName: \"kubernetes.io/projected/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-kube-api-access-jj426\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:19.493211 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:19.493177 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:19.495671 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:19.495639 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:19.615755 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:19.615721 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:19.986582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:19.986557 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk"] Nov 01 04:25:19.988526 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:25:19.988497 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0d6f23_c9f0_4c32_9472_4553b5cf45e6.slice/crio-9687cfac894418c853886edbf3eab7b10e9b5d1d5786ff68f5368eb3a6d48925 WatchSource:0}: Error finding container 9687cfac894418c853886edbf3eab7b10e9b5d1d5786ff68f5368eb3a6d48925: Status 404 returned error can't find the container with id 9687cfac894418c853886edbf3eab7b10e9b5d1d5786ff68f5368eb3a6d48925 Nov 01 04:25:20.570695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:20.570656 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" event={"ID":"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6","Type":"ContainerStarted","Data":"9687cfac894418c853886edbf3eab7b10e9b5d1d5786ff68f5368eb3a6d48925"} Nov 01 04:25:21.578497 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:21.578462 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" event={"ID":"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6","Type":"ContainerStarted","Data":"5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b"} Nov 01 04:25:21.578934 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:21.578545 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:21.690944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:21.690883 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" podStartSLOduration=2.8750176610000002 podStartE2EDuration="3.690869536s" totalImagesPullingTime="815.851875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:25:21.6894413 +0000 UTC m=+2533.278535956" watchObservedRunningTime="2026-11-01 04:25:21.690869536 +0000 UTC m=+2533.279964164" Nov 01 04:25:27.547771 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:27.547741 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:25:27.588521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:27.588495 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:25:29.567652 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:29.567618 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:25:31.578164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:31.578134 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww"] Nov 01 04:25:33.001344 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:25:33.001305 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:25:38.655236 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:38.655192 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t"] Nov 01 04:25:38.754290 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:38.754240 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p"] Nov 01 04:25:43.655982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:43.655941 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" podUID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" containerName="payload-processing" containerID="cri-o://5123ac7d0b17ada5cb98fcc5cd757091ec17f7b6d1156cda238f2305de0257f9" gracePeriod=25 Nov 01 04:25:43.755524 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:43.755468 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" podUID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" containerName="payload-pre-processing" containerID="cri-o://fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077" gracePeriod=25 Nov 01 04:25:46.599204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:46.599166 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerName="main" containerID="cri-o://f45d786929070002ff6f3cb412b10e94090876c0b705fd098dc080486f6d597b" gracePeriod=45 Nov 01 04:25:46.721019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:46.720986 6735 generic.go:356] "Generic (PLEG): container finished" podID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerID="f45d786929070002ff6f3cb412b10e94090876c0b705fd098dc080486f6d597b" exitCode=0 Nov 01 04:25:46.721192 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:46.721055 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" event={"ID":"b7b2dbe6-6923-4d46-82db-6037b289c9d0","Type":"ContainerDied","Data":"f45d786929070002ff6f3cb412b10e94090876c0b705fd098dc080486f6d597b"} Nov 01 04:25:46.873561 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:46.873533 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:25:47.009312 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009280 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tmp-dir\") pod \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " Nov 01 04:25:47.009504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009392 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b2dbe6-6923-4d46-82db-6037b289c9d0-kube-api-access-4ck6s\" (UniqueName: \"kubernetes.io/projected/b7b2dbe6-6923-4d46-82db-6037b289c9d0-kube-api-access-4ck6s\") pod \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " Nov 01 04:25:47.009504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009459 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-dshm\") pod \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " Nov 01 04:25:47.009504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009490 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-model-cache\") pod \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " Nov 01 04:25:47.009685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009529 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-home\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-home\") pod \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " Nov 01 04:25:47.009685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009570 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tls-certs\") pod \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\" (UID: \"b7b2dbe6-6923-4d46-82db-6037b289c9d0\") " Nov 01 04:25:47.009685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009666 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tmp-dir" pod "b7b2dbe6-6923-4d46-82db-6037b289c9d0" (UID: "b7b2dbe6-6923-4d46-82db-6037b289c9d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:25:47.009858 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009772 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-model-cache" pod "b7b2dbe6-6923-4d46-82db-6037b289c9d0" (UID: "b7b2dbe6-6923-4d46-82db-6037b289c9d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:25:47.009918 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.009880 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-home" pod "b7b2dbe6-6923-4d46-82db-6037b289c9d0" (UID: "b7b2dbe6-6923-4d46-82db-6037b289c9d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:25:47.010109 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.010091 6735 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-model-cache\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:25:47.010185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.010113 6735 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-home\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:25:47.010185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.010126 6735 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tmp-dir\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:25:47.012323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.012282 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b2dbe6-6923-4d46-82db-6037b289c9d0-kube-api-access-4ck6s" pod "b7b2dbe6-6923-4d46-82db-6037b289c9d0" (UID: "b7b2dbe6-6923-4d46-82db-6037b289c9d0"). InnerVolumeSpecName "kube-api-access-4ck6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:25:47.012323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.012300 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tls-certs" pod "b7b2dbe6-6923-4d46-82db-6037b289c9d0" (UID: "b7b2dbe6-6923-4d46-82db-6037b289c9d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:25:47.012769 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.012741 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-dshm" pod "b7b2dbe6-6923-4d46-82db-6037b289c9d0" (UID: "b7b2dbe6-6923-4d46-82db-6037b289c9d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:25:47.110936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.110840 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ck6s\" (UniqueName: \"kubernetes.io/projected/b7b2dbe6-6923-4d46-82db-6037b289c9d0-kube-api-access-4ck6s\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:25:47.110936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.110874 6735 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7b2dbe6-6923-4d46-82db-6037b289c9d0-dshm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:25:47.110936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.110890 6735 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7b2dbe6-6923-4d46-82db-6037b289c9d0-tls-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:25:47.728205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.728163 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" event={"ID":"b7b2dbe6-6923-4d46-82db-6037b289c9d0","Type":"ContainerDied","Data":"008d7220ab18c57bc1a0905204cfae8ee49115fc58dc65291e7b1ab6098d0b80"} Nov 01 04:25:47.728685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.728222 6735 scope.go:122] "RemoveContainer" containerID="f45d786929070002ff6f3cb412b10e94090876c0b705fd098dc080486f6d597b" Nov 01 04:25:47.728685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.728186 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww" Nov 01 04:25:47.882481 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.882448 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww"] Nov 01 04:25:47.928730 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:47.928697 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f73b9f/distinct-2-w3-f73b9f-kserve-5d6c85ccb-tpwww"] Nov 01 04:25:48.539815 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:48.539779 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" path="/var/lib/kubelet/pods/b7b2dbe6-6923-4d46-82db-6037b289c9d0/volumes" Nov 01 04:25:49.532475 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:49.532441 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" podUID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:25:50.536935 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:50.536901 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" podUID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:25:55.145209 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:55.145174 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm"] Nov 01 04:25:59.532538 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:25:59.532500 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" podUID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:26:00.536623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:00.536588 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" podUID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:26:01.037001 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:01.036969 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:26:02.166185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:02.166143 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4"] Nov 01 04:26:02.265716 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:02.265680 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds"] Nov 01 04:26:03.001011 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:03.000974 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:26:03.836280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:03.836242 6735 generic.go:356] "Generic (PLEG): container finished" podID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" containerID="5123ac7d0b17ada5cb98fcc5cd757091ec17f7b6d1156cda238f2305de0257f9" exitCode=0 Nov 01 04:26:03.836702 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:03.836381 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" event={"ID":"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0","Type":"ContainerDied","Data":"5123ac7d0b17ada5cb98fcc5cd757091ec17f7b6d1156cda238f2305de0257f9"} Nov 01 04:26:03.954069 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:03.946377 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 04:26:04.024260 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.024236 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 04:26:04.036420 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.036385 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-kube-api-access-tf6wq\" (UniqueName: \"kubernetes.io/projected/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-kube-api-access-tf6wq\") pod \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " Nov 01 04:26:04.036544 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.036498 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-plugins-config-volume\") pod \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\" (UID: \"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0\") " Nov 01 04:26:04.036895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.036872 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-plugins-config-volume" pod "9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" (UID: "9ef761b2-65e2-4c8d-bdb7-13004e3d26c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:26:04.038503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.038481 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-kube-api-access-tf6wq" pod "9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" (UID: "9ef761b2-65e2-4c8d-bdb7-13004e3d26c0"). InnerVolumeSpecName "kube-api-access-tf6wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:26:04.138213 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.138185 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8897b0-1a82-4173-93e8-948fbffe7cf6-kube-api-access-62mpr\" (UniqueName: \"kubernetes.io/projected/cc8897b0-1a82-4173-93e8-948fbffe7cf6-kube-api-access-62mpr\") pod \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " Nov 01 04:26:04.138390 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.138357 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8897b0-1a82-4173-93e8-948fbffe7cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8897b0-1a82-4173-93e8-948fbffe7cf6-plugins-config-volume\") pod \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\" (UID: \"cc8897b0-1a82-4173-93e8-948fbffe7cf6\") " Nov 01 04:26:04.138581 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.138568 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6wq\" (UniqueName: \"kubernetes.io/projected/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-kube-api-access-tf6wq\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:04.138635 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.138582 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:04.138692 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.138671 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8897b0-1a82-4173-93e8-948fbffe7cf6-plugins-config-volume" pod "cc8897b0-1a82-4173-93e8-948fbffe7cf6" (UID: "cc8897b0-1a82-4173-93e8-948fbffe7cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:26:04.140382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.140364 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8897b0-1a82-4173-93e8-948fbffe7cf6-kube-api-access-62mpr" pod "cc8897b0-1a82-4173-93e8-948fbffe7cf6" (UID: "cc8897b0-1a82-4173-93e8-948fbffe7cf6"). InnerVolumeSpecName "kube-api-access-62mpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:26:04.239245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.239217 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8897b0-1a82-4173-93e8-948fbffe7cf6-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:04.239245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.239242 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62mpr\" (UniqueName: \"kubernetes.io/projected/cc8897b0-1a82-4173-93e8-948fbffe7cf6-kube-api-access-62mpr\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:04.843923 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.843841 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" Nov 01 04:26:04.844391 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.843841 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t" event={"ID":"9ef761b2-65e2-4c8d-bdb7-13004e3d26c0","Type":"ContainerDied","Data":"e91ba2fe63ed7c27cd5c0e6ab11100f0bac6a6a50d0184eabdedf0cae1e42eef"} Nov 01 04:26:04.844391 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.843982 6735 scope.go:122] "RemoveContainer" containerID="5123ac7d0b17ada5cb98fcc5cd757091ec17f7b6d1156cda238f2305de0257f9" Nov 01 04:26:04.845817 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.845796 6735 generic.go:356] "Generic (PLEG): container finished" podID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" containerID="fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077" exitCode=0 Nov 01 04:26:04.845898 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.845845 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" Nov 01 04:26:04.845898 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.845882 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" event={"ID":"cc8897b0-1a82-4173-93e8-948fbffe7cf6","Type":"ContainerDied","Data":"fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077"} Nov 01 04:26:04.845982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.845935 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p" event={"ID":"cc8897b0-1a82-4173-93e8-948fbffe7cf6","Type":"ContainerDied","Data":"ca6a48bf6c0eed09d4e0ba67884d581781d19b8a96e2aa590b424db3ecbf50b5"} Nov 01 04:26:04.855641 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.855623 6735 scope.go:122] "RemoveContainer" containerID="fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077" Nov 01 04:26:04.864252 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.864237 6735 scope.go:122] "RemoveContainer" containerID="fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077" Nov 01 04:26:04.864549 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:04.864530 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077\": container with ID starting with fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077 not found: ID does not exist" containerID="fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077" Nov 01 04:26:04.864605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.864557 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077"} err="failed to get container status \"fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077\": rpc error: code = NotFound desc = could not find container \"fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077\": container with ID starting with fc161072cd36001df3417bac015077617177d0ff6dc7edf2e571168121061077 not found: ID does not exist" Nov 01 04:26:04.997461 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:04.997431 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t"] Nov 01 04:26:05.043456 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:05.043427 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f73b9f-759767d4ff-n5q8t"] Nov 01 04:26:05.103884 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:05.103815 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p"] Nov 01 04:26:05.108239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:05.108214 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f73b9f-5499d45f9-7n84p"] Nov 01 04:26:06.525728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:06.525696 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" path="/var/lib/kubelet/pods/9ef761b2-65e2-4c8d-bdb7-13004e3d26c0/volumes" Nov 01 04:26:06.526319 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:06.526288 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" path="/var/lib/kubelet/pods/cc8897b0-1a82-4173-93e8-948fbffe7cf6/volumes" Nov 01 04:26:07.166844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.166802 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" podUID="37b0d047-59ea-4f3e-9d5e-27203656a227" containerName="payload-processing" containerID="cri-o://d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145" gracePeriod=25 Nov 01 04:26:07.266392 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.266340 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" podUID="6a0baee5-5558-4344-ada8-cc90b190bcfa" containerName="payload-pre-processing" containerID="cri-o://5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5" gracePeriod=25 Nov 01 04:26:07.283597 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.283561 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6"] Nov 01 04:26:07.284185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284168 6735 cpu_manager.go:414] "containerMap: removing container" podUID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" containerName="payload-pre-processing" Nov 01 04:26:07.284185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284186 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" containerName="payload-pre-processing" Nov 01 04:26:07.284323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284202 6735 cpu_manager.go:414] "containerMap: removing container" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerName="main" Nov 01 04:26:07.284323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284207 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerName="main" Nov 01 04:26:07.284323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284219 6735 cpu_manager.go:414] "containerMap: removing container" podUID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" containerName="payload-processing" Nov 01 04:26:07.284323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284223 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" containerName="payload-processing" Nov 01 04:26:07.284323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284310 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8897b0-1a82-4173-93e8-948fbffe7cf6" containerName="payload-pre-processing" Nov 01 04:26:07.284323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284317 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b2dbe6-6923-4d46-82db-6037b289c9d0" containerName="main" Nov 01 04:26:07.284323 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.284323 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef761b2-65e2-4c8d-bdb7-13004e3d26c0" containerName="payload-processing" Nov 01 04:26:07.288818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.288799 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.337765 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.337733 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1cdd1d-serving-cert\"" Nov 01 04:26:07.372895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.372863 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6qx\" (UniqueName: \"kubernetes.io/projected/33049e71-d6bc-42ef-8315-da303486267c-kube-api-access-lb6qx\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.373060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.372904 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-metrics-certs\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.373060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.372977 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.373060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.373023 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33049e71-d6bc-42ef-8315-da303486267c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.386261 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.386225 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6"] Nov 01 04:26:07.473765 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.473734 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33049e71-d6bc-42ef-8315-da303486267c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.473925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.473835 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6qx\" (UniqueName: \"kubernetes.io/projected/33049e71-d6bc-42ef-8315-da303486267c-kube-api-access-lb6qx\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.473925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.473860 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-metrics-certs\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.473925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.473907 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.474140 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:07.474069 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-serving-cert: secret "maas-api-e2e-praxis-a-1cdd1d-serving-cert" not found Nov 01 04:26:07.474204 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:07.474143 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls podName:33049e71-d6bc-42ef-8315-da303486267c nodeName:}" failed. No retries permitted until 2026-11-01 04:26:07.974125241 +0000 UTC m=+2579.563219865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls") pod "maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" (UID: "33049e71-d6bc-42ef-8315-da303486267c") : secret "maas-api-e2e-praxis-a-1cdd1d-serving-cert" not found Nov 01 04:26:07.474414 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.474396 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33049e71-d6bc-42ef-8315-da303486267c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.476409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.476386 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-metrics-certs\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.570774 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.570740 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6qx\" (UniqueName: \"kubernetes.io/projected/33049e71-d6bc-42ef-8315-da303486267c-kube-api-access-lb6qx\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.979937 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.979902 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:07.983349 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:07.983318 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:08.201949 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:08.201917 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:08.573540 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:08.573518 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6"] Nov 01 04:26:08.576091 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:26:08.576062 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33049e71_d6bc_42ef_8315_da303486267c.slice/crio-2dd38769efa86bda10363cf5fb4c702329f259c233bccf471ac39f26239f3f1b WatchSource:0}: Error finding container 2dd38769efa86bda10363cf5fb4c702329f259c233bccf471ac39f26239f3f1b: Status 404 returned error can't find the container with id 2dd38769efa86bda10363cf5fb4c702329f259c233bccf471ac39f26239f3f1b Nov 01 04:26:08.883526 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:08.883442 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" event={"ID":"33049e71-d6bc-42ef-8315-da303486267c","Type":"ContainerStarted","Data":"2dd38769efa86bda10363cf5fb4c702329f259c233bccf471ac39f26239f3f1b"} Nov 01 04:26:09.890849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:09.890812 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" event={"ID":"33049e71-d6bc-42ef-8315-da303486267c","Type":"ContainerStarted","Data":"cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4"} Nov 01 04:26:09.891279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:09.891048 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:10.001828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.001775 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" podStartSLOduration=2.180054628 podStartE2EDuration="3.001761565s" totalImagesPullingTime="821.706937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:09.999873476 +0000 UTC m=+2581.588968104" watchObservedRunningTime="2026-11-01 04:26:10.001761565 +0000 UTC m=+2581.590856194" Nov 01 04:26:10.166099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.166000 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" podUID="89dad693-8ce5-4348-a34c-0a3fb36e979e" containerName="main" containerID="cri-o://2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d" gracePeriod=45 Nov 01 04:26:10.434082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.433996 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:26:10.606961 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.606929 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dad693-8ce5-4348-a34c-0a3fb36e979e-kube-api-access-t6pgg\" (UniqueName: \"kubernetes.io/projected/89dad693-8ce5-4348-a34c-0a3fb36e979e-kube-api-access-t6pgg\") pod \"89dad693-8ce5-4348-a34c-0a3fb36e979e\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " Nov 01 04:26:10.607166 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.606970 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-dshm\") pod \"89dad693-8ce5-4348-a34c-0a3fb36e979e\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " Nov 01 04:26:10.607166 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607144 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-tmp-dir\") pod \"89dad693-8ce5-4348-a34c-0a3fb36e979e\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " Nov 01 04:26:10.607293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607211 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-model-cache\") pod \"89dad693-8ce5-4348-a34c-0a3fb36e979e\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " Nov 01 04:26:10.607293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607237 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-home\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-home\") pod \"89dad693-8ce5-4348-a34c-0a3fb36e979e\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " Nov 01 04:26:10.607293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607266 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89dad693-8ce5-4348-a34c-0a3fb36e979e-tls-certs\" (UniqueName: \"kubernetes.io/secret/89dad693-8ce5-4348-a34c-0a3fb36e979e-tls-certs\") pod \"89dad693-8ce5-4348-a34c-0a3fb36e979e\" (UID: \"89dad693-8ce5-4348-a34c-0a3fb36e979e\") " Nov 01 04:26:10.607505 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607421 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-model-cache" pod "89dad693-8ce5-4348-a34c-0a3fb36e979e" (UID: "89dad693-8ce5-4348-a34c-0a3fb36e979e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:26:10.607505 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607438 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-tmp-dir" pod "89dad693-8ce5-4348-a34c-0a3fb36e979e" (UID: "89dad693-8ce5-4348-a34c-0a3fb36e979e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:26:10.607871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607797 6735 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-tmp-dir\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:10.607871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.607829 6735 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-model-cache\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:10.608052 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.608014 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-home" pod "89dad693-8ce5-4348-a34c-0a3fb36e979e" (UID: "89dad693-8ce5-4348-a34c-0a3fb36e979e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:26:10.609398 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.609375 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-dshm" pod "89dad693-8ce5-4348-a34c-0a3fb36e979e" (UID: "89dad693-8ce5-4348-a34c-0a3fb36e979e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:26:10.609832 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.609806 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89dad693-8ce5-4348-a34c-0a3fb36e979e-tls-certs" pod "89dad693-8ce5-4348-a34c-0a3fb36e979e" (UID: "89dad693-8ce5-4348-a34c-0a3fb36e979e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:26:10.609832 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.609825 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dad693-8ce5-4348-a34c-0a3fb36e979e-kube-api-access-t6pgg" pod "89dad693-8ce5-4348-a34c-0a3fb36e979e" (UID: "89dad693-8ce5-4348-a34c-0a3fb36e979e"). InnerVolumeSpecName "kube-api-access-t6pgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:26:10.709327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.709304 6735 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-home\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:10.709471 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.709333 6735 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89dad693-8ce5-4348-a34c-0a3fb36e979e-tls-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:10.709471 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.709343 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6pgg\" (UniqueName: \"kubernetes.io/projected/89dad693-8ce5-4348-a34c-0a3fb36e979e-kube-api-access-t6pgg\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:10.709471 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.709357 6735 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89dad693-8ce5-4348-a34c-0a3fb36e979e-dshm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:10.897514 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.897477 6735 generic.go:356] "Generic (PLEG): container finished" podID="89dad693-8ce5-4348-a34c-0a3fb36e979e" containerID="2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d" exitCode=0 Nov 01 04:26:10.897969 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.897526 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" Nov 01 04:26:10.897969 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.897564 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" event={"ID":"89dad693-8ce5-4348-a34c-0a3fb36e979e","Type":"ContainerDied","Data":"2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d"} Nov 01 04:26:10.897969 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.897605 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm" event={"ID":"89dad693-8ce5-4348-a34c-0a3fb36e979e","Type":"ContainerDied","Data":"370f723d946a56fdc65328532ae8963d45bff95793922975067980f4d881179d"} Nov 01 04:26:10.897969 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.897629 6735 scope.go:122] "RemoveContainer" containerID="2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d" Nov 01 04:26:10.909529 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.909513 6735 scope.go:122] "RemoveContainer" containerID="2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d" Nov 01 04:26:10.909791 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:10.909772 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d\": container with ID starting with 2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d not found: ID does not exist" containerID="2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d" Nov 01 04:26:10.909831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:10.909802 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d"} err="failed to get container status \"2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d\": rpc error: code = NotFound desc = could not find container \"2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d\": container with ID starting with 2a23f3b7c613179e4b1ea81306e07498e1f54547f0e661b548a0b168b24d130d not found: ID does not exist" Nov 01 04:26:11.050800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:11.050723 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm"] Nov 01 04:26:11.097048 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:11.096999 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-162191/distinct-2-w0-162191-kserve-84d4f78c4b-r88hm"] Nov 01 04:26:12.526441 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:12.526402 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89dad693-8ce5-4348-a34c-0a3fb36e979e" path="/var/lib/kubelet/pods/89dad693-8ce5-4348-a34c-0a3fb36e979e/volumes" Nov 01 04:26:15.490283 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:15.490233 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" podUID="37b0d047-59ea-4f3e-9d5e-27203656a227" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:26:15.490768 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:15.490240 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" podUID="6a0baee5-5558-4344-ada8-cc90b190bcfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:26:15.903643 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:15.903566 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:26:25.490586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:25.490539 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" podUID="6a0baee5-5558-4344-ada8-cc90b190bcfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:26:25.490586 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:25.490566 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" podUID="37b0d047-59ea-4f3e-9d5e-27203656a227" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:26:27.434895 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.434829 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 04:26:27.505447 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.505421 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b0d047-59ea-4f3e-9d5e-27203656a227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b0d047-59ea-4f3e-9d5e-27203656a227-plugins-config-volume\") pod \"37b0d047-59ea-4f3e-9d5e-27203656a227\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " Nov 01 04:26:27.505597 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.505467 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b0d047-59ea-4f3e-9d5e-27203656a227-kube-api-access-r2nt7\" (UniqueName: \"kubernetes.io/projected/37b0d047-59ea-4f3e-9d5e-27203656a227-kube-api-access-r2nt7\") pod \"37b0d047-59ea-4f3e-9d5e-27203656a227\" (UID: \"37b0d047-59ea-4f3e-9d5e-27203656a227\") " Nov 01 04:26:27.505787 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.505762 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b0d047-59ea-4f3e-9d5e-27203656a227-plugins-config-volume" pod "37b0d047-59ea-4f3e-9d5e-27203656a227" (UID: "37b0d047-59ea-4f3e-9d5e-27203656a227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:26:27.507677 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.507659 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b0d047-59ea-4f3e-9d5e-27203656a227-kube-api-access-r2nt7" pod "37b0d047-59ea-4f3e-9d5e-27203656a227" (UID: "37b0d047-59ea-4f3e-9d5e-27203656a227"). InnerVolumeSpecName "kube-api-access-r2nt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:26:27.522185 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.522160 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 04:26:27.606631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.606598 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0baee5-5558-4344-ada8-cc90b190bcfa-kube-api-access-vbpws\" (UniqueName: \"kubernetes.io/projected/6a0baee5-5558-4344-ada8-cc90b190bcfa-kube-api-access-vbpws\") pod \"6a0baee5-5558-4344-ada8-cc90b190bcfa\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " Nov 01 04:26:27.606824 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.606773 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0baee5-5558-4344-ada8-cc90b190bcfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0baee5-5558-4344-ada8-cc90b190bcfa-plugins-config-volume\") pod \"6a0baee5-5558-4344-ada8-cc90b190bcfa\" (UID: \"6a0baee5-5558-4344-ada8-cc90b190bcfa\") " Nov 01 04:26:27.607146 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.607122 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0baee5-5558-4344-ada8-cc90b190bcfa-plugins-config-volume" pod "6a0baee5-5558-4344-ada8-cc90b190bcfa" (UID: "6a0baee5-5558-4344-ada8-cc90b190bcfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:26:27.607227 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.607200 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b0d047-59ea-4f3e-9d5e-27203656a227-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:27.607227 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.607222 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2nt7\" (UniqueName: \"kubernetes.io/projected/37b0d047-59ea-4f3e-9d5e-27203656a227-kube-api-access-r2nt7\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:27.608713 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.608693 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0baee5-5558-4344-ada8-cc90b190bcfa-kube-api-access-vbpws" pod "6a0baee5-5558-4344-ada8-cc90b190bcfa" (UID: "6a0baee5-5558-4344-ada8-cc90b190bcfa"). InnerVolumeSpecName "kube-api-access-vbpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:26:27.708687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.708663 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0baee5-5558-4344-ada8-cc90b190bcfa-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:27.708687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.708689 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbpws\" (UniqueName: \"kubernetes.io/projected/6a0baee5-5558-4344-ada8-cc90b190bcfa-kube-api-access-vbpws\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:26:27.993609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.993523 6735 generic.go:356] "Generic (PLEG): container finished" podID="6a0baee5-5558-4344-ada8-cc90b190bcfa" containerID="5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5" exitCode=0 Nov 01 04:26:27.993609 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.993570 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" Nov 01 04:26:27.993849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.993616 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" event={"ID":"6a0baee5-5558-4344-ada8-cc90b190bcfa","Type":"ContainerDied","Data":"5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5"} Nov 01 04:26:27.993849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.993672 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds" event={"ID":"6a0baee5-5558-4344-ada8-cc90b190bcfa","Type":"ContainerDied","Data":"67e6911abb05be137047e4623faeeae71c19e257bbdf36be73743c4c5e6ea11c"} Nov 01 04:26:27.993849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.993701 6735 scope.go:122] "RemoveContainer" containerID="5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5" Nov 01 04:26:27.995060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.995014 6735 generic.go:356] "Generic (PLEG): container finished" podID="37b0d047-59ea-4f3e-9d5e-27203656a227" containerID="d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145" exitCode=0 Nov 01 04:26:27.995060 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.995057 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" event={"ID":"37b0d047-59ea-4f3e-9d5e-27203656a227","Type":"ContainerDied","Data":"d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145"} Nov 01 04:26:27.995266 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.995080 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" Nov 01 04:26:27.995266 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:27.995086 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4" event={"ID":"37b0d047-59ea-4f3e-9d5e-27203656a227","Type":"ContainerDied","Data":"12c97bf2bba0443f3368047f067e9280839d51df116f3581c8b1efbd04a84606"} Nov 01 04:26:28.003550 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.003534 6735 scope.go:122] "RemoveContainer" containerID="5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5" Nov 01 04:26:28.003809 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:28.003784 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5\": container with ID starting with 5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5 not found: ID does not exist" containerID="5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5" Nov 01 04:26:28.003916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.003816 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5"} err="failed to get container status \"5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5\": rpc error: code = NotFound desc = could not find container \"5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5\": container with ID starting with 5e62c5c20f455bfc127656a591fe8e110d62afeb187627f4cb9c064fc5a5dae5 not found: ID does not exist" Nov 01 04:26:28.003916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.003840 6735 scope.go:122] "RemoveContainer" containerID="d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145" Nov 01 04:26:28.012299 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.012279 6735 scope.go:122] "RemoveContainer" containerID="d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145" Nov 01 04:26:28.012540 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:28.012522 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145\": container with ID starting with d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145 not found: ID does not exist" containerID="d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145" Nov 01 04:26:28.012600 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.012546 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145"} err="failed to get container status \"d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145\": rpc error: code = NotFound desc = could not find container \"d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145\": container with ID starting with d121b9632096f70ff999ff0328545c743d15cc35a133db5c28d7d638306f0145 not found: ID does not exist" Nov 01 04:26:28.150295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.150261 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds"] Nov 01 04:26:28.196827 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.196783 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-162191-594cffb5c9-9ngds"] Nov 01 04:26:28.255898 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.255820 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4"] Nov 01 04:26:28.261948 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.261924 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-162191-7b85fb9855-mfgf4"] Nov 01 04:26:28.527240 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.527162 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b0d047-59ea-4f3e-9d5e-27203656a227" path="/var/lib/kubelet/pods/37b0d047-59ea-4f3e-9d5e-27203656a227/volumes" Nov 01 04:26:28.527624 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:28.527605 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0baee5-5558-4344-ada8-cc90b190bcfa" path="/var/lib/kubelet/pods/6a0baee5-5558-4344-ada8-cc90b190bcfa/volumes" Nov 01 04:26:33.000906 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:26:33.000867 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:26:37.023788 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.023742 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs"] Nov 01 04:26:37.024479 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024458 6735 cpu_manager.go:414] "containerMap: removing container" podUID="6a0baee5-5558-4344-ada8-cc90b190bcfa" containerName="payload-pre-processing" Nov 01 04:26:37.024577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024482 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0baee5-5558-4344-ada8-cc90b190bcfa" containerName="payload-pre-processing" Nov 01 04:26:37.024577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024496 6735 cpu_manager.go:414] "containerMap: removing container" podUID="89dad693-8ce5-4348-a34c-0a3fb36e979e" containerName="main" Nov 01 04:26:37.024577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024503 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dad693-8ce5-4348-a34c-0a3fb36e979e" containerName="main" Nov 01 04:26:37.024577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024558 6735 cpu_manager.go:414] "containerMap: removing container" podUID="37b0d047-59ea-4f3e-9d5e-27203656a227" containerName="payload-processing" Nov 01 04:26:37.024577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024566 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b0d047-59ea-4f3e-9d5e-27203656a227" containerName="payload-processing" Nov 01 04:26:37.024828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024673 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b0d047-59ea-4f3e-9d5e-27203656a227" containerName="payload-processing" Nov 01 04:26:37.024828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024693 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0baee5-5558-4344-ada8-cc90b190bcfa" containerName="payload-pre-processing" Nov 01 04:26:37.024828 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.024704 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dad693-8ce5-4348-a34c-0a3fb36e979e" containerName="main" Nov 01 04:26:37.028053 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.028018 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.117057 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.116736 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f24878-3d90-4f0d-97f4-01b8907714ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.117057 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.116899 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rw6\" (UniqueName: \"kubernetes.io/projected/c6f24878-3d90-4f0d-97f4-01b8907714ef-kube-api-access-44rw6\") pod \"payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.123317 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.123290 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f7545f-dockercfg-rnkxq\"" Nov 01 04:26:37.124151 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.123648 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f7545f\"" Nov 01 04:26:37.132374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.132325 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs"] Nov 01 04:26:37.217750 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.217700 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f24878-3d90-4f0d-97f4-01b8907714ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.217936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.217819 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rw6\" (UniqueName: \"kubernetes.io/projected/c6f24878-3d90-4f0d-97f4-01b8907714ef-kube-api-access-44rw6\") pod \"payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.218449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.218429 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f24878-3d90-4f0d-97f4-01b8907714ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.275745 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.275660 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rw6\" (UniqueName: \"kubernetes.io/projected/c6f24878-3d90-4f0d-97f4-01b8907714ef-kube-api-access-44rw6\") pod \"payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.340996 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.340959 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:37.712381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:37.712358 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs"] Nov 01 04:26:37.714830 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:26:37.714797 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f24878_3d90_4f0d_97f4_01b8907714ef.slice/crio-98f83f4ba09115a89cf4b249ea64ed7e308395f62e76be0df652e63f65acc40f WatchSource:0}: Error finding container 98f83f4ba09115a89cf4b249ea64ed7e308395f62e76be0df652e63f65acc40f: Status 404 returned error can't find the container with id 98f83f4ba09115a89cf4b249ea64ed7e308395f62e76be0df652e63f65acc40f Nov 01 04:26:38.052291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:38.052198 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" event={"ID":"c6f24878-3d90-4f0d-97f4-01b8907714ef","Type":"ContainerStarted","Data":"6265293b1c688358e30403a00ae19f2480a66b3ab1493817b4aa4b5b6d2a5cbb"} Nov 01 04:26:38.052291 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:38.052254 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" event={"ID":"c6f24878-3d90-4f0d-97f4-01b8907714ef","Type":"ContainerStarted","Data":"98f83f4ba09115a89cf4b249ea64ed7e308395f62e76be0df652e63f65acc40f"} Nov 01 04:26:38.052707 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:38.052295 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:26:38.163332 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:38.163279 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" podStartSLOduration=2.163264184 podStartE2EDuration="2.163264184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:26:38.16155535 +0000 UTC m=+2609.750649979" watchObservedRunningTime="2026-11-01 04:26:38.163264184 +0000 UTC m=+2609.752358812" Nov 01 04:26:48.031381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:48.031347 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:26:49.060896 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:26:49.060870 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:27:03.001208 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:03.001171 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:27:05.587598 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.587563 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg"] Nov 01 04:27:05.591472 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.591447 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.640239 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.640210 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-804014-serving-cert\"" Nov 01 04:27:05.648341 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.648314 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.648521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.648497 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7pml\" (UniqueName: \"kubernetes.io/projected/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-kube-api-access-x7pml\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.648588 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.648542 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-metrics-certs\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.648737 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.648720 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.688549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.688517 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg"] Nov 01 04:27:05.750405 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.750373 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.750563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.750424 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.750563 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.750472 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7pml\" (UniqueName: \"kubernetes.io/projected/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-kube-api-access-x7pml\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.750655 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:05.750569 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-serving-cert: secret "maas-api-e2e-praxis-b-804014-serving-cert" not found Nov 01 04:27:05.750655 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:05.750641 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls podName:1abc408e-54fa-4dd7-ab70-f7e3cbd03554 nodeName:}" failed. No retries permitted until 2026-11-01 04:27:06.25062011 +0000 UTC m=+2637.839714730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls") pod "maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" (UID: "1abc408e-54fa-4dd7-ab70-f7e3cbd03554") : secret "maas-api-e2e-praxis-b-804014-serving-cert" not found Nov 01 04:27:05.750766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.750566 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-metrics-certs\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.751080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.751057 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.752998 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.752980 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-metrics-certs\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:05.810572 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:05.810536 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7pml\" (UniqueName: \"kubernetes.io/projected/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-kube-api-access-x7pml\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:06.259878 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:06.259839 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:06.262385 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:06.262362 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls\") pod \"maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:06.505067 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:06.505015 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:06.882108 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:06.882082 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg"] Nov 01 04:27:06.884374 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:27:06.884345 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abc408e_54fa_4dd7_ab70_f7e3cbd03554.slice/crio-d68537bcb7870f4398e45968976ffae4f4d204111dbdb77d74076b79b4b578a4 WatchSource:0}: Error finding container d68537bcb7870f4398e45968976ffae4f4d204111dbdb77d74076b79b4b578a4: Status 404 returned error can't find the container with id d68537bcb7870f4398e45968976ffae4f4d204111dbdb77d74076b79b4b578a4 Nov 01 04:27:07.212777 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:07.212743 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" event={"ID":"1abc408e-54fa-4dd7-ab70-f7e3cbd03554","Type":"ContainerStarted","Data":"d68537bcb7870f4398e45968976ffae4f4d204111dbdb77d74076b79b4b578a4"} Nov 01 04:27:08.220297 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:08.220256 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" event={"ID":"1abc408e-54fa-4dd7-ab70-f7e3cbd03554","Type":"ContainerStarted","Data":"4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c"} Nov 01 04:27:08.220727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:08.220356 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:08.331689 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:08.331638 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" podStartSLOduration=2.501544838 podStartE2EDuration="3.331623816s" totalImagesPullingTime="830.078978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:27:08.329932665 +0000 UTC m=+2639.919027288" watchObservedRunningTime="2026-11-01 04:27:08.331623816 +0000 UTC m=+2639.920718444" Nov 01 04:27:14.231681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:14.231642 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:27:15.852473 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:15.852435 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49"] Nov 01 04:27:15.852925 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:15.852660 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" podUID="a957d6e5-260b-42fc-a274-4e564503c3ab" containerName="maas-api" containerID="cri-o://d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c" gracePeriod=30 Nov 01 04:27:16.101682 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.101658 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 04:27:16.267183 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.267147 6735 generic.go:356] "Generic (PLEG): container finished" podID="a957d6e5-260b-42fc-a274-4e564503c3ab" containerID="d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c" exitCode=0 Nov 01 04:27:16.267374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.267199 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" Nov 01 04:27:16.267374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.267239 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" event={"ID":"a957d6e5-260b-42fc-a274-4e564503c3ab","Type":"ContainerDied","Data":"d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c"} Nov 01 04:27:16.267374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.267294 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49" event={"ID":"a957d6e5-260b-42fc-a274-4e564503c3ab","Type":"ContainerDied","Data":"28f109f90dc0b6a0ed68f101c31dbcf3e761c1ee107b7da0e2e05c4a5c32848a"} Nov 01 04:27:16.267374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.267325 6735 scope.go:122] "RemoveContainer" containerID="d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c" Nov 01 04:27:16.276851 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.276831 6735 scope.go:122] "RemoveContainer" containerID="d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c" Nov 01 04:27:16.277141 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.277122 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a957d6e5-260b-42fc-a274-4e564503c3ab-kube-api-access-6kv75\" (UniqueName: \"kubernetes.io/projected/a957d6e5-260b-42fc-a274-4e564503c3ab-kube-api-access-6kv75\") pod \"a957d6e5-260b-42fc-a274-4e564503c3ab\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " Nov 01 04:27:16.277250 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:16.277141 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c\": container with ID starting with d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c not found: ID does not exist" containerID="d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c" Nov 01 04:27:16.277250 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.277174 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c"} err="failed to get container status \"d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c\": rpc error: code = NotFound desc = could not find container \"d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c\": container with ID starting with d57b3461825d131dde074b4f71f358fb6ee1046b103c5b975d3455a5adfb549c not found: ID does not exist" Nov 01 04:27:16.277250 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.277189 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls\") pod \"a957d6e5-260b-42fc-a274-4e564503c3ab\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " Nov 01 04:27:16.277439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.277289 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-metrics-certs\") pod \"a957d6e5-260b-42fc-a274-4e564503c3ab\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " Nov 01 04:27:16.277439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.277356 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a957d6e5-260b-42fc-a274-4e564503c3ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a957d6e5-260b-42fc-a274-4e564503c3ab-openshift-service-ca\") pod \"a957d6e5-260b-42fc-a274-4e564503c3ab\" (UID: \"a957d6e5-260b-42fc-a274-4e564503c3ab\") " Nov 01 04:27:16.278091 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.278067 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a957d6e5-260b-42fc-a274-4e564503c3ab-openshift-service-ca" pod "a957d6e5-260b-42fc-a274-4e564503c3ab" (UID: "a957d6e5-260b-42fc-a274-4e564503c3ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:27:16.279287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.279264 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls" pod "a957d6e5-260b-42fc-a274-4e564503c3ab" (UID: "a957d6e5-260b-42fc-a274-4e564503c3ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:27:16.279387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.279360 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a957d6e5-260b-42fc-a274-4e564503c3ab-kube-api-access-6kv75" pod "a957d6e5-260b-42fc-a274-4e564503c3ab" (UID: "a957d6e5-260b-42fc-a274-4e564503c3ab"). InnerVolumeSpecName "kube-api-access-6kv75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:27:16.279444 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.279381 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-metrics-certs" pod "a957d6e5-260b-42fc-a274-4e564503c3ab" (UID: "a957d6e5-260b-42fc-a274-4e564503c3ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:27:16.379074 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.379008 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:16.379308 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.379290 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a957d6e5-260b-42fc-a274-4e564503c3ab-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:16.379440 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.379421 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a957d6e5-260b-42fc-a274-4e564503c3ab-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:16.379530 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.379520 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kv75\" (UniqueName: \"kubernetes.io/projected/a957d6e5-260b-42fc-a274-4e564503c3ab-kube-api-access-6kv75\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:16.681223 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.681188 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49"] Nov 01 04:27:16.684675 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:16.684648 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-162191-7c7c58645-pcl49"] Nov 01 04:27:17.452381 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:17.452346 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:27:18.526269 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:18.526234 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a957d6e5-260b-42fc-a274-4e564503c3ab" path="/var/lib/kubelet/pods/a957d6e5-260b-42fc-a274-4e564503c3ab/volumes" Nov 01 04:27:25.629209 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:25.629174 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd"] Nov 01 04:27:25.629710 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:25.629392 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" containerID="cri-o://79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53" gracePeriod=30 Nov 01 04:27:29.540802 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.540766 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg"] Nov 01 04:27:29.541584 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.541560 6735 cpu_manager.go:414] "containerMap: removing container" podUID="a957d6e5-260b-42fc-a274-4e564503c3ab" containerName="maas-api" Nov 01 04:27:29.541584 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.541585 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a957d6e5-260b-42fc-a274-4e564503c3ab" containerName="maas-api" Nov 01 04:27:29.541719 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.541692 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="a957d6e5-260b-42fc-a274-4e564503c3ab" containerName="maas-api" Nov 01 04:27:29.546418 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.546398 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.598228 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.598190 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f7545f-serving-cert\"" Nov 01 04:27:29.629177 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.629139 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.629384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.629217 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5148b38d-b561-49b1-bce9-1a999749d468-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.629445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.629407 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwqn\" (UniqueName: \"kubernetes.io/projected/5148b38d-b561-49b1-bce9-1a999749d468-kube-api-access-tzwqn\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.629506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.629463 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-metrics-certs\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.644171 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.644135 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg"] Nov 01 04:27:29.730554 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.730520 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5148b38d-b561-49b1-bce9-1a999749d468-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.730743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.730576 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwqn\" (UniqueName: \"kubernetes.io/projected/5148b38d-b561-49b1-bce9-1a999749d468-kube-api-access-tzwqn\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.730743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.730615 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-metrics-certs\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.730743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.730663 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.730855 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:29.730814 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-serving-cert: secret "maas-api-e2e-praxis-b-f7545f-serving-cert" not found Nov 01 04:27:29.730908 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:29.730896 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls podName:5148b38d-b561-49b1-bce9-1a999749d468 nodeName:}" failed. No retries permitted until 2026-11-01 04:27:30.230874538 +0000 UTC m=+2661.819969144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls") pod "maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" (UID: "5148b38d-b561-49b1-bce9-1a999749d468") : secret "maas-api-e2e-praxis-b-f7545f-serving-cert" not found Nov 01 04:27:29.731358 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.731332 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5148b38d-b561-49b1-bce9-1a999749d468-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.733344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.733327 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-metrics-certs\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:29.789248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:29.789214 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwqn\" (UniqueName: \"kubernetes.io/projected/5148b38d-b561-49b1-bce9-1a999749d468-kube-api-access-tzwqn\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:30.237288 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.237249 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:30.245685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.245647 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:30.460103 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.460058 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:30.829096 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.829068 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg"] Nov 01 04:27:30.832019 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:27:30.831983 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5148b38d_b561_49b1_bce9_1a999749d468.slice/crio-48f6877fc4d5a48011957bd14ee3d10dcfecd60b8081732b8aa19b54b0031375 WatchSource:0}: Error finding container 48f6877fc4d5a48011957bd14ee3d10dcfecd60b8081732b8aa19b54b0031375: Status 404 returned error can't find the container with id 48f6877fc4d5a48011957bd14ee3d10dcfecd60b8081732b8aa19b54b0031375 Nov 01 04:27:30.907623 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.907595 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 04:27:30.944655 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944623 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-token\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-token\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.944813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944676 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-credential-socket\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.944813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944741 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-certs\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.944813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944765 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-data\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.944813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944806 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7401861-17b9-48d2-b5bd-705c2368ed67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7401861-17b9-48d2-b5bd-705c2368ed67-istiod-ca-cert\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.944968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944878 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-kube-api-access-jntmp\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-kube-api-access-jntmp\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.944968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944930 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-socket\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.945119 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.944983 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-podinfo\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.945119 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945021 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-certs" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:30.945119 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945084 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-data" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:30.945119 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945069 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-envoy\") pod \"e7401861-17b9-48d2-b5bd-705c2368ed67\" (UID: \"e7401861-17b9-48d2-b5bd-705c2368ed67\") " Nov 01 04:27:30.945318 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945280 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-socket" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:30.945368 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945338 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7401861-17b9-48d2-b5bd-705c2368ed67-istiod-ca-cert" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:27:30.945429 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945368 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-credential-socket" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:30.945577 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945561 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:30.945831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945583 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:30.945831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945594 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:30.945831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945602 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:30.945831 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.945610 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7401861-17b9-48d2-b5bd-705c2368ed67-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:30.947344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.947316 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-envoy" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:30.947456 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.947339 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-kube-api-access-jntmp" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "kube-api-access-jntmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:27:30.947456 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.947374 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-token" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:27:30.947600 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:30.947581 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-podinfo" pod "e7401861-17b9-48d2-b5bd-705c2368ed67" (UID: "e7401861-17b9-48d2-b5bd-705c2368ed67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:27:31.047292 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.047251 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jntmp\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-kube-api-access-jntmp\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:31.047292 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.047285 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:31.047292 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.047294 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:31.047540 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.047303 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7401861-17b9-48d2-b5bd-705c2368ed67-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:31.350115 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.350066 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" event={"ID":"5148b38d-b561-49b1-bce9-1a999749d468","Type":"ContainerStarted","Data":"48f6877fc4d5a48011957bd14ee3d10dcfecd60b8081732b8aa19b54b0031375"} Nov 01 04:27:31.351504 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.351477 6735 generic.go:356] "Generic (PLEG): container finished" podID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerID="79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53" exitCode=0 Nov 01 04:27:31.351640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.351529 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" Nov 01 04:27:31.351640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.351531 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" event={"ID":"e7401861-17b9-48d2-b5bd-705c2368ed67","Type":"ContainerDied","Data":"79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53"} Nov 01 04:27:31.351770 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.351663 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd" event={"ID":"e7401861-17b9-48d2-b5bd-705c2368ed67","Type":"ContainerDied","Data":"c3ee8d0ecc004e8af8b994c5191668e2cad2f480366de501aa29a55550ffe8ec"} Nov 01 04:27:31.351770 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.351702 6735 scope.go:122] "RemoveContainer" containerID="79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53" Nov 01 04:27:31.361813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.361792 6735 scope.go:122] "RemoveContainer" containerID="79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53" Nov 01 04:27:31.362083 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:31.362060 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53\": container with ID starting with 79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53 not found: ID does not exist" containerID="79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53" Nov 01 04:27:31.362145 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.362092 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53"} err="failed to get container status \"79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53\": rpc error: code = NotFound desc = could not find container \"79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53\": container with ID starting with 79e37a58c534fb3990e703cffbb28938f5c135bdc1ac54d2c51a5a47c6018c53 not found: ID does not exist" Nov 01 04:27:31.396101 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.396069 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj"] Nov 01 04:27:31.396336 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.396314 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" containerID="cri-o://05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990" gracePeriod=30 Nov 01 04:27:31.646522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.646441 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd"] Nov 01 04:27:31.650164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:31.650134 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-162191-openshift-default-699bc96769-jnmsd"] Nov 01 04:27:32.359887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:32.359847 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" event={"ID":"5148b38d-b561-49b1-bce9-1a999749d468","Type":"ContainerStarted","Data":"9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750"} Nov 01 04:27:32.360314 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:32.359988 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:32.469825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:32.469768 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" podStartSLOduration=2.57999627 podStartE2EDuration="3.469753034s" totalImagesPullingTime="889.756764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:27:32.469603692 +0000 UTC m=+2664.058698321" watchObservedRunningTime="2026-11-01 04:27:32.469753034 +0000 UTC m=+2664.058847664" Nov 01 04:27:32.529087 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:32.529028 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" path="/var/lib/kubelet/pods/e7401861-17b9-48d2-b5bd-705c2368ed67/volumes" Nov 01 04:27:33.001808 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:33.001776 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:27:36.661592 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.661531 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 04:27:36.710181 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710136 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4583095-5413-4a7f-9515-f0cab6c7b374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4583095-5413-4a7f-9515-f0cab6c7b374-istiod-ca-cert\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710362 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710200 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-credential-socket\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710362 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710238 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-data\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710362 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710270 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-socket\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710362 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710293 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-envoy\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710362 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710320 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-certs\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710620 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710381 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-podinfo\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710620 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710435 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-kube-api-access-nfxtl\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-kube-api-access-nfxtl\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710620 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710511 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-token\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-token\") pod \"e4583095-5413-4a7f-9515-f0cab6c7b374\" (UID: \"e4583095-5413-4a7f-9515-f0cab6c7b374\") " Nov 01 04:27:36.710783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710633 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-credential-socket" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:36.710783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710650 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-data" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:36.710783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710691 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-socket" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:36.710783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710722 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4583095-5413-4a7f-9515-f0cab6c7b374-istiod-ca-cert" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:27:36.710783 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.710718 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-certs" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:36.711075 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.711017 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4583095-5413-4a7f-9515-f0cab6c7b374-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.711137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.711074 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.711137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.711088 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.711137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.711101 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.711137 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.711114 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.712813 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.712790 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-token" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:27:36.712903 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.712866 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-podinfo" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:27:36.713303 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.713277 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-envoy" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:27:36.713303 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.713287 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-kube-api-access-nfxtl" pod "e4583095-5413-4a7f-9515-f0cab6c7b374" (UID: "e4583095-5413-4a7f-9515-f0cab6c7b374"). InnerVolumeSpecName "kube-api-access-nfxtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:27:36.812205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.812169 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.812205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.812198 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.812205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.812207 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxtl\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-kube-api-access-nfxtl\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:36.812205 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:36.812217 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4583095-5413-4a7f-9515-f0cab6c7b374-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:27:37.388676 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.388641 6735 generic.go:356] "Generic (PLEG): container finished" podID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerID="05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990" exitCode=0 Nov 01 04:27:37.388898 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.388698 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" Nov 01 04:27:37.388898 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.388737 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" event={"ID":"e4583095-5413-4a7f-9515-f0cab6c7b374","Type":"ContainerDied","Data":"05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990"} Nov 01 04:27:37.388898 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.388804 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj" event={"ID":"e4583095-5413-4a7f-9515-f0cab6c7b374","Type":"ContainerDied","Data":"843c92e341e0bd9e414da3e407070b6ede7bfb6daa79440e3475552e1de06144"} Nov 01 04:27:37.388898 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.388834 6735 scope.go:122] "RemoveContainer" containerID="05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990" Nov 01 04:27:37.399007 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.398985 6735 scope.go:122] "RemoveContainer" containerID="333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf" Nov 01 04:27:37.407634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.407616 6735 scope.go:122] "RemoveContainer" containerID="05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990" Nov 01 04:27:37.407930 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:37.407909 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990\": container with ID starting with 05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990 not found: ID does not exist" containerID="05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990" Nov 01 04:27:37.407979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.407941 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990"} err="failed to get container status \"05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990\": rpc error: code = NotFound desc = could not find container \"05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990\": container with ID starting with 05e2ca7e18c7b7cdb91c2f3de6e174d78c4ccbbb84b3a248e5d1a4bd18b53990 not found: ID does not exist" Nov 01 04:27:37.407979 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.407959 6735 scope.go:122] "RemoveContainer" containerID="333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf" Nov 01 04:27:37.408252 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:27:37.408232 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf\": container with ID starting with 333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf not found: ID does not exist" containerID="333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf" Nov 01 04:27:37.408315 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.408262 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf"} err="failed to get container status \"333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf\": rpc error: code = NotFound desc = could not find container \"333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf\": container with ID starting with 333b8df0b2c2ae0f0c076c73ff1562a84f9a02999e6e4e07686677bf2f50bfaf not found: ID does not exist" Nov 01 04:27:37.590849 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.590815 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj"] Nov 01 04:27:37.636721 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:37.636689 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f73b9f-openshift-default-7cb5c77d8f-p92hj"] Nov 01 04:27:38.370665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:38.370637 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:27:38.528161 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:38.528122 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" path="/var/lib/kubelet/pods/e4583095-5413-4a7f-9515-f0cab6c7b374/volumes" Nov 01 04:27:39.912645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:27:39.912600 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:28:03.001523 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:28:03.001484 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:28:09.481104 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481072 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb"] Nov 01 04:28:09.481595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481581 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" Nov 01 04:28:09.481640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481596 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" Nov 01 04:28:09.481640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481613 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" Nov 01 04:28:09.481640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481618 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" Nov 01 04:28:09.481640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481637 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" Nov 01 04:28:09.481766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481642 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" Nov 01 04:28:09.481766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481728 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" Nov 01 04:28:09.481766 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.481740 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7401861-17b9-48d2-b5bd-705c2368ed67" containerName="istio-proxy" Nov 01 04:28:09.484907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.484888 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.533944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.533910 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b8319b-serving-cert\"" Nov 01 04:28:09.541057 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.541003 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb"] Nov 01 04:28:09.564230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.564203 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a190b3-d99e-46f1-841c-9aa77bda2ec5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.564428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.564247 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvvk\" (UniqueName: \"kubernetes.io/projected/41a190b3-d99e-46f1-841c-9aa77bda2ec5-kube-api-access-ltvvk\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.564428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.564391 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-metrics-certs\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.564509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.564464 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.665675 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.665638 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a190b3-d99e-46f1-841c-9aa77bda2ec5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.665887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.665696 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvvk\" (UniqueName: \"kubernetes.io/projected/41a190b3-d99e-46f1-841c-9aa77bda2ec5-kube-api-access-ltvvk\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.665887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.665753 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-metrics-certs\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.665887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.665821 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.666110 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:28:09.665980 6735 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-serving-cert: secret "maas-api-e2e-praxis-b-b8319b-serving-cert" not found Nov 01 04:28:09.666110 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:28:09.666083 6735 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls podName:41a190b3-d99e-46f1-841c-9aa77bda2ec5 nodeName:}" failed. No retries permitted until 2026-11-01 04:28:10.166060181 +0000 UTC m=+2701.755154789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls") pod "maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" (UID: "41a190b3-d99e-46f1-841c-9aa77bda2ec5") : secret "maas-api-e2e-praxis-b-b8319b-serving-cert" not found Nov 01 04:28:09.666330 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.666308 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a190b3-d99e-46f1-841c-9aa77bda2ec5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.668307 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.668283 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-metrics-certs\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.726653 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.726617 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvvk\" (UniqueName: \"kubernetes.io/projected/41a190b3-d99e-46f1-841c-9aa77bda2ec5-kube-api-access-ltvvk\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:09.863538 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.863464 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:28:09.877395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:09.877372 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:28:10.170745 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:10.170667 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:10.173322 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:10.173300 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:10.398716 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:10.398680 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:10.767833 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:10.767807 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb"] Nov 01 04:28:10.770190 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:28:10.770157 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a190b3_d99e_46f1_841c_9aa77bda2ec5.slice/crio-040680758c068344d7af940f361fb7b7d3ddafa4128e7d2c61d5b3e83d16e0d8 WatchSource:0}: Error finding container 040680758c068344d7af940f361fb7b7d3ddafa4128e7d2c61d5b3e83d16e0d8: Status 404 returned error can't find the container with id 040680758c068344d7af940f361fb7b7d3ddafa4128e7d2c61d5b3e83d16e0d8 Nov 01 04:28:11.589532 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:11.589491 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" event={"ID":"41a190b3-d99e-46f1-841c-9aa77bda2ec5","Type":"ContainerStarted","Data":"040680758c068344d7af940f361fb7b7d3ddafa4128e7d2c61d5b3e83d16e0d8"} Nov 01 04:28:12.596212 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:12.596170 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" event={"ID":"41a190b3-d99e-46f1-841c-9aa77bda2ec5","Type":"ContainerStarted","Data":"993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817"} Nov 01 04:28:12.596705 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:12.596412 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:12.707520 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:12.707466 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" podStartSLOduration=2.793508375 podStartE2EDuration="3.707450377s" totalImagesPullingTime="913.942002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:28:12.705508308 +0000 UTC m=+2704.294602937" watchObservedRunningTime="2026-11-01 04:28:12.707450377 +0000 UTC m=+2704.296545006" Nov 01 04:28:18.608276 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:18.608208 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:28:24.224444 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:24.224408 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:28:27.766654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:27.766615 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt"] Nov 01 04:28:27.863595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:27.863560 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x"] Nov 01 04:28:32.767516 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:32.767463 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" podUID="fe4778da-864c-47de-b518-a01e669b5b45" containerName="payload-processing" containerID="cri-o://8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891" gracePeriod=25 Nov 01 04:28:32.863866 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:32.863825 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" podUID="1a45e4da-0cd9-4fed-b37e-aff6ea285f91" containerName="payload-pre-processing" containerID="cri-o://9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a" gracePeriod=25 Nov 01 04:28:33.005823 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:28:33.005788 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:28:33.116430 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.116400 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:28:33.151699 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.151676 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:28:33.213724 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.213696 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-plugins-config-volume\") pod \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " Nov 01 04:28:33.213860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.213777 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4778da-864c-47de-b518-a01e669b5b45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4778da-864c-47de-b518-a01e669b5b45-plugins-config-volume\") pod \"fe4778da-864c-47de-b518-a01e669b5b45\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " Nov 01 04:28:33.213860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.213798 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-kube-api-access-wmw9m\" (UniqueName: \"kubernetes.io/projected/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-kube-api-access-wmw9m\") pod \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\" (UID: \"1a45e4da-0cd9-4fed-b37e-aff6ea285f91\") " Nov 01 04:28:33.213933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.213867 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4778da-864c-47de-b518-a01e669b5b45-kube-api-access-9pp2l\" (UniqueName: \"kubernetes.io/projected/fe4778da-864c-47de-b518-a01e669b5b45-kube-api-access-9pp2l\") pod \"fe4778da-864c-47de-b518-a01e669b5b45\" (UID: \"fe4778da-864c-47de-b518-a01e669b5b45\") " Nov 01 04:28:33.214114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.214083 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-plugins-config-volume" pod "1a45e4da-0cd9-4fed-b37e-aff6ea285f91" (UID: "1a45e4da-0cd9-4fed-b37e-aff6ea285f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:28:33.214224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.214174 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:33.214224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.214174 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4778da-864c-47de-b518-a01e669b5b45-plugins-config-volume" pod "fe4778da-864c-47de-b518-a01e669b5b45" (UID: "fe4778da-864c-47de-b518-a01e669b5b45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:28:33.216098 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.216077 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4778da-864c-47de-b518-a01e669b5b45-kube-api-access-9pp2l" pod "fe4778da-864c-47de-b518-a01e669b5b45" (UID: "fe4778da-864c-47de-b518-a01e669b5b45"). InnerVolumeSpecName "kube-api-access-9pp2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:28:33.216170 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.216109 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-kube-api-access-wmw9m" pod "1a45e4da-0cd9-4fed-b37e-aff6ea285f91" (UID: "1a45e4da-0cd9-4fed-b37e-aff6ea285f91"). InnerVolumeSpecName "kube-api-access-wmw9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:28:33.315138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.315057 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4778da-864c-47de-b518-a01e669b5b45-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:33.315138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.315084 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmw9m\" (UniqueName: \"kubernetes.io/projected/1a45e4da-0cd9-4fed-b37e-aff6ea285f91-kube-api-access-wmw9m\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:33.315138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.315098 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pp2l\" (UniqueName: \"kubernetes.io/projected/fe4778da-864c-47de-b518-a01e669b5b45-kube-api-access-9pp2l\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:33.709438 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.709406 6735 generic.go:356] "Generic (PLEG): container finished" podID="1a45e4da-0cd9-4fed-b37e-aff6ea285f91" containerID="9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a" exitCode=0 Nov 01 04:28:33.709632 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.709458 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" Nov 01 04:28:33.709632 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.709487 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" event={"ID":"1a45e4da-0cd9-4fed-b37e-aff6ea285f91","Type":"ContainerDied","Data":"9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a"} Nov 01 04:28:33.709632 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.709544 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x" event={"ID":"1a45e4da-0cd9-4fed-b37e-aff6ea285f91","Type":"ContainerDied","Data":"e7815cb649923d80248169b03e1469627c1141ddaeea8fd2bc8d2c0c05ae2347"} Nov 01 04:28:33.709632 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.709578 6735 scope.go:122] "RemoveContainer" containerID="9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a" Nov 01 04:28:33.710981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.710962 6735 generic.go:356] "Generic (PLEG): container finished" podID="fe4778da-864c-47de-b518-a01e669b5b45" containerID="8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891" exitCode=0 Nov 01 04:28:33.711092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.711010 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" event={"ID":"fe4778da-864c-47de-b518-a01e669b5b45","Type":"ContainerDied","Data":"8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891"} Nov 01 04:28:33.711092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.711015 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" Nov 01 04:28:33.711092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.711067 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt" event={"ID":"fe4778da-864c-47de-b518-a01e669b5b45","Type":"ContainerDied","Data":"fb3f8caadfbdb4567a4486cf99b26d1e7ff71334df7c864fad3a24a24ecaab12"} Nov 01 04:28:33.721247 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.721232 6735 scope.go:122] "RemoveContainer" containerID="9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a" Nov 01 04:28:33.721533 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:28:33.721513 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a\": container with ID starting with 9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a not found: ID does not exist" containerID="9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a" Nov 01 04:28:33.721582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.721540 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a"} err="failed to get container status \"9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a\": rpc error: code = NotFound desc = could not find container \"9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a\": container with ID starting with 9d53cf04d4c71e79691922fd7ec946c068271e600535219a8a0d74b2ae5e6f8a not found: ID does not exist" Nov 01 04:28:33.721582 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.721556 6735 scope.go:122] "RemoveContainer" containerID="8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891" Nov 01 04:28:33.729816 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.729799 6735 scope.go:122] "RemoveContainer" containerID="8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891" Nov 01 04:28:33.730100 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:28:33.730081 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891\": container with ID starting with 8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891 not found: ID does not exist" containerID="8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891" Nov 01 04:28:33.730148 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.730106 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891"} err="failed to get container status \"8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891\": rpc error: code = NotFound desc = could not find container \"8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891\": container with ID starting with 8f11275abad386a8e4a3b0e525db3df393cdd1d036deb7a4fb4cb7e1c2e71891 not found: ID does not exist" Nov 01 04:28:33.863968 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.863933 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt"] Nov 01 04:28:33.910164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:33.910131 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-550b8a-76f96777b5-rkbnt"] Nov 01 04:28:34.012277 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:34.012191 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x"] Nov 01 04:28:34.016088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:34.016062 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-550b8a-64cd4c79f-z244x"] Nov 01 04:28:34.524971 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:34.524939 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a45e4da-0cd9-4fed-b37e-aff6ea285f91" path="/var/lib/kubelet/pods/1a45e4da-0cd9-4fed-b37e-aff6ea285f91/volumes" Nov 01 04:28:34.525372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:34.525358 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4778da-864c-47de-b518-a01e669b5b45" path="/var/lib/kubelet/pods/fe4778da-864c-47de-b518-a01e669b5b45/volumes" Nov 01 04:28:50.099228 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.099193 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj"] Nov 01 04:28:50.099791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.099448 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" podUID="435e600c-65ea-412b-b429-1da4c5ccc826" containerName="maas-api" containerID="cri-o://fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422" gracePeriod=30 Nov 01 04:28:50.380120 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.380087 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:28:50.508372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.506823 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435e600c-65ea-412b-b429-1da4c5ccc826-kube-api-access-qxpm2\" (UniqueName: \"kubernetes.io/projected/435e600c-65ea-412b-b429-1da4c5ccc826-kube-api-access-qxpm2\") pod \"435e600c-65ea-412b-b429-1da4c5ccc826\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " Nov 01 04:28:50.508372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.506880 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435e600c-65ea-412b-b429-1da4c5ccc826-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e600c-65ea-412b-b429-1da4c5ccc826-openshift-service-ca\") pod \"435e600c-65ea-412b-b429-1da4c5ccc826\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " Nov 01 04:28:50.508372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.506924 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls\") pod \"435e600c-65ea-412b-b429-1da4c5ccc826\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " Nov 01 04:28:50.508372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.506963 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-metrics-certs\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-metrics-certs\") pod \"435e600c-65ea-412b-b429-1da4c5ccc826\" (UID: \"435e600c-65ea-412b-b429-1da4c5ccc826\") " Nov 01 04:28:50.508372 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.508098 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435e600c-65ea-412b-b429-1da4c5ccc826-openshift-service-ca" pod "435e600c-65ea-412b-b429-1da4c5ccc826" (UID: "435e600c-65ea-412b-b429-1da4c5ccc826"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:28:50.510873 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.510841 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435e600c-65ea-412b-b429-1da4c5ccc826-kube-api-access-qxpm2" pod "435e600c-65ea-412b-b429-1da4c5ccc826" (UID: "435e600c-65ea-412b-b429-1da4c5ccc826"). InnerVolumeSpecName "kube-api-access-qxpm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:28:50.513588 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.513545 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-metrics-certs" pod "435e600c-65ea-412b-b429-1da4c5ccc826" (UID: "435e600c-65ea-412b-b429-1da4c5ccc826"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:28:50.514292 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.514224 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls" pod "435e600c-65ea-412b-b429-1da4c5ccc826" (UID: "435e600c-65ea-412b-b429-1da4c5ccc826"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:28:50.615002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.614712 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:50.615002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.614754 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxpm2\" (UniqueName: \"kubernetes.io/projected/435e600c-65ea-412b-b429-1da4c5ccc826-kube-api-access-qxpm2\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:50.615002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.614772 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/435e600c-65ea-412b-b429-1da4c5ccc826-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:50.615002 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.614787 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/435e600c-65ea-412b-b429-1da4c5ccc826-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:28:50.809480 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.809445 6735 generic.go:356] "Generic (PLEG): container finished" podID="435e600c-65ea-412b-b429-1da4c5ccc826" containerID="fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422" exitCode=0 Nov 01 04:28:50.809654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.809494 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" Nov 01 04:28:50.809654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.809499 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" event={"ID":"435e600c-65ea-412b-b429-1da4c5ccc826","Type":"ContainerDied","Data":"fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422"} Nov 01 04:28:50.809654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.809540 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj" event={"ID":"435e600c-65ea-412b-b429-1da4c5ccc826","Type":"ContainerDied","Data":"9f061fdcd21d6d794eb85f6ae7bb637234e9869b1d7c5729d97ce997c6a0175f"} Nov 01 04:28:50.809654 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.809561 6735 scope.go:122] "RemoveContainer" containerID="fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422" Nov 01 04:28:50.818852 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.818824 6735 scope.go:122] "RemoveContainer" containerID="fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422" Nov 01 04:28:50.819130 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:28:50.819105 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422\": container with ID starting with fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422 not found: ID does not exist" containerID="fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422" Nov 01 04:28:50.819240 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.819134 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422"} err="failed to get container status \"fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422\": rpc error: code = NotFound desc = could not find container \"fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422\": container with ID starting with fb6db6d4efa6cb561d4c65443f447f9c4c29934c97a7b5eb5aca872011bf3422 not found: ID does not exist" Nov 01 04:28:50.962743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:50.962702 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj"] Nov 01 04:28:51.009153 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:51.009122 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-550b8a-64b48ccc6f-qvgrj"] Nov 01 04:28:52.531963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:52.531924 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435e600c-65ea-412b-b429-1da4c5ccc826" path="/var/lib/kubelet/pods/435e600c-65ea-412b-b429-1da4c5ccc826/volumes" Nov 01 04:28:58.597991 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:28:58.597956 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:29:03.001586 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:29:03.001555 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:29:30.533678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:30.533646 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj"] Nov 01 04:29:30.635710 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:30.635674 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt"] Nov 01 04:29:33.000564 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:29:33.000520 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:29:35.534116 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.534077 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" podUID="03003767-c63c-42f5-84b1-8956917491b0" containerName="payload-processing" containerID="cri-o://d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1" gracePeriod=25 Nov 01 04:29:35.636121 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.636086 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" podUID="2f724760-4640-41a2-b471-fe60addef235" containerName="payload-pre-processing" containerID="cri-o://5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73" gracePeriod=25 Nov 01 04:29:35.799289 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.799223 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:29:35.875063 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.875018 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03003767-c63c-42f5-84b1-8956917491b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03003767-c63c-42f5-84b1-8956917491b0-plugins-config-volume\") pod \"03003767-c63c-42f5-84b1-8956917491b0\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " Nov 01 04:29:35.875258 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.875179 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03003767-c63c-42f5-84b1-8956917491b0-kube-api-access-bsxtg\" (UniqueName: \"kubernetes.io/projected/03003767-c63c-42f5-84b1-8956917491b0-kube-api-access-bsxtg\") pod \"03003767-c63c-42f5-84b1-8956917491b0\" (UID: \"03003767-c63c-42f5-84b1-8956917491b0\") " Nov 01 04:29:35.875413 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.875391 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03003767-c63c-42f5-84b1-8956917491b0-plugins-config-volume" pod "03003767-c63c-42f5-84b1-8956917491b0" (UID: "03003767-c63c-42f5-84b1-8956917491b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:29:35.875545 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.875528 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03003767-c63c-42f5-84b1-8956917491b0-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:35.877893 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.877875 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03003767-c63c-42f5-84b1-8956917491b0-kube-api-access-bsxtg" pod "03003767-c63c-42f5-84b1-8956917491b0" (UID: "03003767-c63c-42f5-84b1-8956917491b0"). InnerVolumeSpecName "kube-api-access-bsxtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:29:35.906468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.906444 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:29:35.976647 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.976613 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f724760-4640-41a2-b471-fe60addef235-kube-api-access-qmd2d\" (UniqueName: \"kubernetes.io/projected/2f724760-4640-41a2-b471-fe60addef235-kube-api-access-qmd2d\") pod \"2f724760-4640-41a2-b471-fe60addef235\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " Nov 01 04:29:35.976815 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.976675 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f724760-4640-41a2-b471-fe60addef235-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f724760-4640-41a2-b471-fe60addef235-plugins-config-volume\") pod \"2f724760-4640-41a2-b471-fe60addef235\" (UID: \"2f724760-4640-41a2-b471-fe60addef235\") " Nov 01 04:29:35.977085 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.977025 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f724760-4640-41a2-b471-fe60addef235-plugins-config-volume" pod "2f724760-4640-41a2-b471-fe60addef235" (UID: "2f724760-4640-41a2-b471-fe60addef235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:29:35.977211 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.977115 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxtg\" (UniqueName: \"kubernetes.io/projected/03003767-c63c-42f5-84b1-8956917491b0-kube-api-access-bsxtg\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:35.978860 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:35.978844 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f724760-4640-41a2-b471-fe60addef235-kube-api-access-qmd2d" pod "2f724760-4640-41a2-b471-fe60addef235" (UID: "2f724760-4640-41a2-b471-fe60addef235"). InnerVolumeSpecName "kube-api-access-qmd2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:29:36.071503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.071421 6735 generic.go:356] "Generic (PLEG): container finished" podID="2f724760-4640-41a2-b471-fe60addef235" containerID="5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73" exitCode=0 Nov 01 04:29:36.071503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.071470 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" Nov 01 04:29:36.071703 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.071510 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" event={"ID":"2f724760-4640-41a2-b471-fe60addef235","Type":"ContainerDied","Data":"5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73"} Nov 01 04:29:36.071703 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.071569 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt" event={"ID":"2f724760-4640-41a2-b471-fe60addef235","Type":"ContainerDied","Data":"453beca7de3d900bc9bb1fb2f086478a2586e93a803126864e8121644bbd9152"} Nov 01 04:29:36.071703 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.071599 6735 scope.go:122] "RemoveContainer" containerID="5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73" Nov 01 04:29:36.072855 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.072835 6735 generic.go:356] "Generic (PLEG): container finished" podID="03003767-c63c-42f5-84b1-8956917491b0" containerID="d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1" exitCode=0 Nov 01 04:29:36.072967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.072875 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" Nov 01 04:29:36.072967 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.072929 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" event={"ID":"03003767-c63c-42f5-84b1-8956917491b0","Type":"ContainerDied","Data":"d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1"} Nov 01 04:29:36.073094 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.073000 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj" event={"ID":"03003767-c63c-42f5-84b1-8956917491b0","Type":"ContainerDied","Data":"5be72f031be4f331ca9e59dd2a7c7d852dc0f5c21d1cb88ece995ad3eb7e57b0"} Nov 01 04:29:36.078166 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.078114 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmd2d\" (UniqueName: \"kubernetes.io/projected/2f724760-4640-41a2-b471-fe60addef235-kube-api-access-qmd2d\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:36.078166 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.078142 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f724760-4640-41a2-b471-fe60addef235-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:36.085975 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.085957 6735 scope.go:122] "RemoveContainer" containerID="5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73" Nov 01 04:29:36.086277 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:29:36.086259 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73\": container with ID starting with 5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73 not found: ID does not exist" containerID="5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73" Nov 01 04:29:36.086331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.086285 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73"} err="failed to get container status \"5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73\": rpc error: code = NotFound desc = could not find container \"5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73\": container with ID starting with 5641031eb06e842669f09c14264660e26ebba912953a6e9331c0eae92eb01b73 not found: ID does not exist" Nov 01 04:29:36.086331 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.086301 6735 scope.go:122] "RemoveContainer" containerID="d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1" Nov 01 04:29:36.094489 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.094470 6735 scope.go:122] "RemoveContainer" containerID="d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1" Nov 01 04:29:36.094739 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:29:36.094722 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1\": container with ID starting with d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1 not found: ID does not exist" containerID="d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1" Nov 01 04:29:36.094794 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.094746 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1"} err="failed to get container status \"d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1\": rpc error: code = NotFound desc = could not find container \"d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1\": container with ID starting with d7793394ce716b54ffe0f41155a0011396ef90b73b98d10200b7b5e414fbe2b1 not found: ID does not exist" Nov 01 04:29:36.224500 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.224466 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt"] Nov 01 04:29:36.228341 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.228316 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1cdd1d-ff69dc895-x6mlt"] Nov 01 04:29:36.331164 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.331074 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj"] Nov 01 04:29:36.334345 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.334320 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1cdd1d-59f8d69dfc-5wmrj"] Nov 01 04:29:36.525157 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.525121 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03003767-c63c-42f5-84b1-8956917491b0" path="/var/lib/kubelet/pods/03003767-c63c-42f5-84b1-8956917491b0/volumes" Nov 01 04:29:36.525602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:36.525582 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f724760-4640-41a2-b471-fe60addef235" path="/var/lib/kubelet/pods/2f724760-4640-41a2-b471-fe60addef235/volumes" Nov 01 04:29:48.350585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.350549 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6"] Nov 01 04:29:48.351107 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.350741 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" podUID="33049e71-d6bc-42ef-8315-da303486267c" containerName="maas-api" containerID="cri-o://cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4" gracePeriod=30 Nov 01 04:29:48.623784 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.623754 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:29:48.702798 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.702768 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33049e71-d6bc-42ef-8315-da303486267c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33049e71-d6bc-42ef-8315-da303486267c-openshift-service-ca\") pod \"33049e71-d6bc-42ef-8315-da303486267c\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " Nov 01 04:29:48.702962 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.702861 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-metrics-certs\") pod \"33049e71-d6bc-42ef-8315-da303486267c\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " Nov 01 04:29:48.702962 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.702915 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls\") pod \"33049e71-d6bc-42ef-8315-da303486267c\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " Nov 01 04:29:48.703098 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.702983 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33049e71-d6bc-42ef-8315-da303486267c-kube-api-access-lb6qx\" (UniqueName: \"kubernetes.io/projected/33049e71-d6bc-42ef-8315-da303486267c-kube-api-access-lb6qx\") pod \"33049e71-d6bc-42ef-8315-da303486267c\" (UID: \"33049e71-d6bc-42ef-8315-da303486267c\") " Nov 01 04:29:48.703207 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.703184 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33049e71-d6bc-42ef-8315-da303486267c-openshift-service-ca" pod "33049e71-d6bc-42ef-8315-da303486267c" (UID: "33049e71-d6bc-42ef-8315-da303486267c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:29:48.703406 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.703386 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33049e71-d6bc-42ef-8315-da303486267c-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:48.705224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.705200 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-metrics-certs" pod "33049e71-d6bc-42ef-8315-da303486267c" (UID: "33049e71-d6bc-42ef-8315-da303486267c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:29:48.705224 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.705219 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls" pod "33049e71-d6bc-42ef-8315-da303486267c" (UID: "33049e71-d6bc-42ef-8315-da303486267c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:29:48.705365 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.705257 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33049e71-d6bc-42ef-8315-da303486267c-kube-api-access-lb6qx" pod "33049e71-d6bc-42ef-8315-da303486267c" (UID: "33049e71-d6bc-42ef-8315-da303486267c"). InnerVolumeSpecName "kube-api-access-lb6qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:29:48.804567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.804528 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:48.804567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.804561 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb6qx\" (UniqueName: \"kubernetes.io/projected/33049e71-d6bc-42ef-8315-da303486267c-kube-api-access-lb6qx\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:48.804567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:48.804574 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33049e71-d6bc-42ef-8315-da303486267c-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:49.145568 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.145532 6735 generic.go:356] "Generic (PLEG): container finished" podID="33049e71-d6bc-42ef-8315-da303486267c" containerID="cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4" exitCode=0 Nov 01 04:29:49.145770 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.145591 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" Nov 01 04:29:49.145770 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.145626 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" event={"ID":"33049e71-d6bc-42ef-8315-da303486267c","Type":"ContainerDied","Data":"cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4"} Nov 01 04:29:49.145770 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.145695 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6" event={"ID":"33049e71-d6bc-42ef-8315-da303486267c","Type":"ContainerDied","Data":"2dd38769efa86bda10363cf5fb4c702329f259c233bccf471ac39f26239f3f1b"} Nov 01 04:29:49.145770 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.145719 6735 scope.go:122] "RemoveContainer" containerID="cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4" Nov 01 04:29:49.155818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.155802 6735 scope.go:122] "RemoveContainer" containerID="cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4" Nov 01 04:29:49.156106 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:29:49.156087 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4\": container with ID starting with cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4 not found: ID does not exist" containerID="cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4" Nov 01 04:29:49.156156 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.156118 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4"} err="failed to get container status \"cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4\": rpc error: code = NotFound desc = could not find container \"cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4\": container with ID starting with cfb4fe2fbb88c8ec56c3be9f2b5af2e0d9f3503d9a26d5e5339f595dc88137a4 not found: ID does not exist" Nov 01 04:29:49.299519 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.299482 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6"] Nov 01 04:29:49.307079 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:49.306997 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1cdd1d-79594777bc-wfhg6"] Nov 01 04:29:50.526074 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:50.526024 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33049e71-d6bc-42ef-8315-da303486267c" path="/var/lib/kubelet/pods/33049e71-d6bc-42ef-8315-da303486267c/volumes" Nov 01 04:29:52.887470 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:52.887435 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh"] Nov 01 04:29:52.887867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:52.887639 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" containerID="cri-o://77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d" gracePeriod=30 Nov 01 04:29:54.746394 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:54.746352 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg"] Nov 01 04:29:54.746964 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:54.746616 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" podUID="1abc408e-54fa-4dd7-ab70-f7e3cbd03554" containerName="maas-api" containerID="cri-o://4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c" gracePeriod=30 Nov 01 04:29:55.023697 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.023672 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:29:55.079773 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.079747 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls\") pod \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " Nov 01 04:29:55.079936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.079791 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-openshift-service-ca\") pod \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " Nov 01 04:29:55.079936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.079810 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-kube-api-access-x7pml\" (UniqueName: \"kubernetes.io/projected/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-kube-api-access-x7pml\") pod \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " Nov 01 04:29:55.079936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.079901 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-metrics-certs\") pod \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\" (UID: \"1abc408e-54fa-4dd7-ab70-f7e3cbd03554\") " Nov 01 04:29:55.080215 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.080191 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-openshift-service-ca" pod "1abc408e-54fa-4dd7-ab70-f7e3cbd03554" (UID: "1abc408e-54fa-4dd7-ab70-f7e3cbd03554"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:29:55.080300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.080280 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:55.082114 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.082086 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls" pod "1abc408e-54fa-4dd7-ab70-f7e3cbd03554" (UID: "1abc408e-54fa-4dd7-ab70-f7e3cbd03554"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:29:55.082231 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.082146 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-metrics-certs" pod "1abc408e-54fa-4dd7-ab70-f7e3cbd03554" (UID: "1abc408e-54fa-4dd7-ab70-f7e3cbd03554"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:29:55.082287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.082251 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-kube-api-access-x7pml" pod "1abc408e-54fa-4dd7-ab70-f7e3cbd03554" (UID: "1abc408e-54fa-4dd7-ab70-f7e3cbd03554"). InnerVolumeSpecName "kube-api-access-x7pml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:29:55.180933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.180894 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:55.180933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.180923 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7pml\" (UniqueName: \"kubernetes.io/projected/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-kube-api-access-x7pml\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:55.180933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.180942 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abc408e-54fa-4dd7-ab70-f7e3cbd03554-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:55.181345 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.181322 6735 generic.go:356] "Generic (PLEG): container finished" podID="1abc408e-54fa-4dd7-ab70-f7e3cbd03554" containerID="4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c" exitCode=0 Nov 01 04:29:55.181409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.181381 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" event={"ID":"1abc408e-54fa-4dd7-ab70-f7e3cbd03554","Type":"ContainerDied","Data":"4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c"} Nov 01 04:29:55.181409 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.181383 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" Nov 01 04:29:55.181554 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.181422 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg" event={"ID":"1abc408e-54fa-4dd7-ab70-f7e3cbd03554","Type":"ContainerDied","Data":"d68537bcb7870f4398e45968976ffae4f4d204111dbdb77d74076b79b4b578a4"} Nov 01 04:29:55.181554 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.181470 6735 scope.go:122] "RemoveContainer" containerID="4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c" Nov 01 04:29:55.191500 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.191482 6735 scope.go:122] "RemoveContainer" containerID="4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c" Nov 01 04:29:55.191782 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:29:55.191758 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c\": container with ID starting with 4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c not found: ID does not exist" containerID="4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c" Nov 01 04:29:55.191843 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.191793 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c"} err="failed to get container status \"4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c\": rpc error: code = NotFound desc = could not find container \"4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c\": container with ID starting with 4417a02d0ddcb403669d3724d63331dc8c2339ad182021ef9d4d7cecdb99381c not found: ID does not exist" Nov 01 04:29:55.277798 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.277712 6735 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-j94nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:29:55.277798 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.277783 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:29:55.335367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.335318 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg"] Nov 01 04:29:55.338835 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:55.338810 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-804014-65b8f45f7b-6d6tg"] Nov 01 04:29:56.525388 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:56.525354 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abc408e-54fa-4dd7-ab70-f7e3cbd03554" path="/var/lib/kubelet/pods/1abc408e-54fa-4dd7-ab70-f7e3cbd03554/volumes" Nov 01 04:29:58.133680 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.133654 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:29:58.147818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.147798 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:29:58.197778 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.197709 6735 generic.go:356] "Generic (PLEG): container finished" podID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerID="77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d" exitCode=0 Nov 01 04:29:58.197778 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.197759 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" Nov 01 04:29:58.197982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.197804 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" event={"ID":"e1003142-0404-4993-a9b0-8ddd5cb38f40","Type":"ContainerDied","Data":"77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d"} Nov 01 04:29:58.197982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.197866 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh" event={"ID":"e1003142-0404-4993-a9b0-8ddd5cb38f40","Type":"ContainerDied","Data":"f652dfd68aec88c118c5bca9e57e9c667adb966d6d9e2afd489ff3328ccf4c4a"} Nov 01 04:29:58.197982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.197900 6735 scope.go:122] "RemoveContainer" containerID="77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d" Nov 01 04:29:58.206854 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.206834 6735 scope.go:122] "RemoveContainer" containerID="77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d" Nov 01 04:29:58.207134 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:29:58.207113 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d\": container with ID starting with 77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d not found: ID does not exist" containerID="77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d" Nov 01 04:29:58.207212 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.207146 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d"} err="failed to get container status \"77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d\": rpc error: code = NotFound desc = could not find container \"77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d\": container with ID starting with 77d7413d09327d6552302999ffb9b37dfb492143e6b0e8fb332e5662fff0da4d not found: ID does not exist" Nov 01 04:29:58.211457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211435 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-envoy\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211519 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211472 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-credential-socket\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211561 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211534 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-data\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211604 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211580 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-kube-api-access-2k2qz\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-kube-api-access-2k2qz\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211650 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211636 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-certs\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211669 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-token\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-token\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211744 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211724 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-podinfo\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211756 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-socket\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.211800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211779 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-credential-socket" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:29:58.211800 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211793 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1003142-0404-4993-a9b0-8ddd5cb38f40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1003142-0404-4993-a9b0-8ddd5cb38f40-istiod-ca-cert\") pod \"e1003142-0404-4993-a9b0-8ddd5cb38f40\" (UID: \"e1003142-0404-4993-a9b0-8ddd5cb38f40\") " Nov 01 04:29:58.212019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.211999 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-socket" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:29:58.212198 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.212146 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-data" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:29:58.212382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.212348 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1003142-0404-4993-a9b0-8ddd5cb38f40-istiod-ca-cert" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:29:58.212727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.212466 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-certs" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:29:58.212727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.212453 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.212727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.212698 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.212727 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.212717 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.214127 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.214080 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-envoy" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:29:58.214223 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.214186 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-kube-api-access-2k2qz" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "kube-api-access-2k2qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:29:58.214383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.214317 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-token" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:29:58.214550 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.214527 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-podinfo" pod "e1003142-0404-4993-a9b0-8ddd5cb38f40" (UID: "e1003142-0404-4993-a9b0-8ddd5cb38f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:29:58.313679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.313642 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1003142-0404-4993-a9b0-8ddd5cb38f40-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.313679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.313672 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.313679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.313683 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2qz\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-kube-api-access-2k2qz\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.313910 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.313693 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1003142-0404-4993-a9b0-8ddd5cb38f40-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.313910 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.313701 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.313910 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.313709 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1003142-0404-4993-a9b0-8ddd5cb38f40-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:29:58.613862 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.613829 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh"] Nov 01 04:29:58.617365 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:29:58.617330 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-j94nh"] Nov 01 04:30:00.201603 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.201564 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk"] Nov 01 04:30:00.202077 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202061 6735 cpu_manager.go:414] "containerMap: removing container" podUID="33049e71-d6bc-42ef-8315-da303486267c" containerName="maas-api" Nov 01 04:30:00.202133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202079 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33049e71-d6bc-42ef-8315-da303486267c" containerName="maas-api" Nov 01 04:30:00.202133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202095 6735 cpu_manager.go:414] "containerMap: removing container" podUID="fe4778da-864c-47de-b518-a01e669b5b45" containerName="payload-processing" Nov 01 04:30:00.202133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202100 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4778da-864c-47de-b518-a01e669b5b45" containerName="payload-processing" Nov 01 04:30:00.202133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202123 6735 cpu_manager.go:414] "containerMap: removing container" podUID="03003767-c63c-42f5-84b1-8956917491b0" containerName="payload-processing" Nov 01 04:30:00.202133 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202128 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03003767-c63c-42f5-84b1-8956917491b0" containerName="payload-processing" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202136 6735 cpu_manager.go:414] "containerMap: removing container" podUID="2f724760-4640-41a2-b471-fe60addef235" containerName="payload-pre-processing" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202140 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f724760-4640-41a2-b471-fe60addef235" containerName="payload-pre-processing" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202154 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1abc408e-54fa-4dd7-ab70-f7e3cbd03554" containerName="maas-api" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202158 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abc408e-54fa-4dd7-ab70-f7e3cbd03554" containerName="maas-api" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202165 6735 cpu_manager.go:414] "containerMap: removing container" podUID="435e600c-65ea-412b-b429-1da4c5ccc826" containerName="maas-api" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202169 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435e600c-65ea-412b-b429-1da4c5ccc826" containerName="maas-api" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202176 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1a45e4da-0cd9-4fed-b37e-aff6ea285f91" containerName="payload-pre-processing" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202180 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a45e4da-0cd9-4fed-b37e-aff6ea285f91" containerName="payload-pre-processing" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202188 6735 cpu_manager.go:414] "containerMap: removing container" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202193 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202255 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abc408e-54fa-4dd7-ab70-f7e3cbd03554" containerName="maas-api" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202263 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="435e600c-65ea-412b-b429-1da4c5ccc826" containerName="maas-api" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202268 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a45e4da-0cd9-4fed-b37e-aff6ea285f91" containerName="payload-pre-processing" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202274 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" containerName="istio-proxy" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202282 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4583095-5413-4a7f-9515-f0cab6c7b374" containerName="istio-proxy" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202289 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="33049e71-d6bc-42ef-8315-da303486267c" containerName="maas-api" Nov 01 04:30:00.202294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202295 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4778da-864c-47de-b518-a01e669b5b45" containerName="payload-processing" Nov 01 04:30:00.202793 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202303 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="03003767-c63c-42f5-84b1-8956917491b0" containerName="payload-processing" Nov 01 04:30:00.202793 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.202310 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f724760-4640-41a2-b471-fe60addef235" containerName="payload-pre-processing" Nov 01 04:30:00.207029 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.207004 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8"] Nov 01 04:30:00.207190 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.207120 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.212190 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.212169 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.255969 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.255936 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\"" Nov 01 04:30:00.321230 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.321193 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8"] Nov 01 04:30:00.333852 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.333821 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhrd\" (UniqueName: \"kubernetes.io/projected/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-kube-api-access-jrhrd\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.334010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.333877 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcgj\" (UniqueName: \"kubernetes.io/projected/1e44b907-7cec-40a8-87f4-3f697a247d25-kube-api-access-5xcgj\") pod \"maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.334010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.333909 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44b907-7cec-40a8-87f4-3f697a247d25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.334010 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.333966 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.390608 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.390574 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk"] Nov 01 04:30:00.435245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.435213 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44b907-7cec-40a8-87f4-3f697a247d25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.435245 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.435248 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.435523 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.435329 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhrd\" (UniqueName: \"kubernetes.io/projected/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-kube-api-access-jrhrd\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.435523 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.435370 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcgj\" (UniqueName: \"kubernetes.io/projected/1e44b907-7cec-40a8-87f4-3f697a247d25-kube-api-access-5xcgj\") pod \"maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.435918 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.435893 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44b907-7cec-40a8-87f4-3f697a247d25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.436016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.435988 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.495400 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.495366 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcgj\" (UniqueName: \"kubernetes.io/projected/1e44b907-7cec-40a8-87f4-3f697a247d25-kube-api-access-5xcgj\") pod \"maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.495606 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.495581 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhrd\" (UniqueName: \"kubernetes.io/projected/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-kube-api-access-jrhrd\") pod \"maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.521491 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.521467 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:00.526615 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.526589 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1003142-0404-4993-a9b0-8ddd5cb38f40" path="/var/lib/kubelet/pods/e1003142-0404-4993-a9b0-8ddd5cb38f40/volumes" Nov 01 04:30:00.528383 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.528364 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:00.889069 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.888974 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk"] Nov 01 04:30:00.897551 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:30:00.897517 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401bdad5_fe06_4bf6_b145_5ae0a9af9e90.slice/crio-50979c5ff25e8feb94b62867704845e97ef1bcaa2dc6e016b48827b925174a0f WatchSource:0}: Error finding container 50979c5ff25e8feb94b62867704845e97ef1bcaa2dc6e016b48827b925174a0f: Status 404 returned error can't find the container with id 50979c5ff25e8feb94b62867704845e97ef1bcaa2dc6e016b48827b925174a0f Nov 01 04:30:00.935944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:00.935918 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8"] Nov 01 04:30:00.937803 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:30:00.937780 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e44b907_7cec_40a8_87f4_3f697a247d25.slice/crio-51a561068e99bb8c5c9b2f62182bb0227a9cdc0ac07be0fa9d1c0df2e484e916 WatchSource:0}: Error finding container 51a561068e99bb8c5c9b2f62182bb0227a9cdc0ac07be0fa9d1c0df2e484e916: Status 404 returned error can't find the container with id 51a561068e99bb8c5c9b2f62182bb0227a9cdc0ac07be0fa9d1c0df2e484e916 Nov 01 04:30:01.217830 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:01.217792 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" event={"ID":"401bdad5-fe06-4bf6-b145-5ae0a9af9e90","Type":"ContainerStarted","Data":"0ef953ea366de869e61ef2646bf04e660fedc5d3fcd9d372cbc1b58982e8cc2f"} Nov 01 04:30:01.218294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:01.217856 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" event={"ID":"401bdad5-fe06-4bf6-b145-5ae0a9af9e90","Type":"ContainerStarted","Data":"50979c5ff25e8feb94b62867704845e97ef1bcaa2dc6e016b48827b925174a0f"} Nov 01 04:30:01.219069 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:01.219022 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" event={"ID":"1e44b907-7cec-40a8-87f4-3f697a247d25","Type":"ContainerStarted","Data":"51a561068e99bb8c5c9b2f62182bb0227a9cdc0ac07be0fa9d1c0df2e484e916"} Nov 01 04:30:01.327636 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:01.327588 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" podStartSLOduration=1.327574676 podStartE2EDuration="1.327574676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:01.326974595 +0000 UTC m=+2812.916069225" watchObservedRunningTime="2026-11-01 04:30:01.327574676 +0000 UTC m=+2812.916669283" Nov 01 04:30:02.225301 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:02.225264 6735 generic.go:356] "Generic (PLEG): container finished" podID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerID="eb8c3bbc8da37f12c10d76b3dcc9708acaba77ea22e39c4c900e7fe84bef9fa1" exitCode=60 Nov 01 04:30:02.225793 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:02.225358 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" event={"ID":"1e44b907-7cec-40a8-87f4-3f697a247d25","Type":"ContainerDied","Data":"eb8c3bbc8da37f12c10d76b3dcc9708acaba77ea22e39c4c900e7fe84bef9fa1"} Nov 01 04:30:02.225793 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:02.225627 6735 scope.go:122] "RemoveContainer" containerID="eb8c3bbc8da37f12c10d76b3dcc9708acaba77ea22e39c4c900e7fe84bef9fa1" Nov 01 04:30:02.227123 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:02.227102 6735 generic.go:356] "Generic (PLEG): container finished" podID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerID="0ef953ea366de869e61ef2646bf04e660fedc5d3fcd9d372cbc1b58982e8cc2f" exitCode=60 Nov 01 04:30:02.227265 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:02.227156 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" event={"ID":"401bdad5-fe06-4bf6-b145-5ae0a9af9e90","Type":"ContainerDied","Data":"0ef953ea366de869e61ef2646bf04e660fedc5d3fcd9d372cbc1b58982e8cc2f"} Nov 01 04:30:02.227410 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:02.227391 6735 scope.go:122] "RemoveContainer" containerID="0ef953ea366de869e61ef2646bf04e660fedc5d3fcd9d372cbc1b58982e8cc2f" Nov 01 04:30:03.002206 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:03.002166 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:30:03.234845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:03.234806 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" event={"ID":"1e44b907-7cec-40a8-87f4-3f697a247d25","Type":"ContainerStarted","Data":"bbd8428ffb40157a54f499f070cf38ef751317654287dfb03bd8b6ffb8eeb764"} Nov 01 04:30:03.240232 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:03.240204 6735 generic.go:356] "Generic (PLEG): container finished" podID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerID="d8815523cbebd9e0c26ffb9350086ec65687ca54e1a05b1e5e12eae956b65c3a" exitCode=60 Nov 01 04:30:03.240380 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:03.240264 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" event={"ID":"401bdad5-fe06-4bf6-b145-5ae0a9af9e90","Type":"ContainerDied","Data":"d8815523cbebd9e0c26ffb9350086ec65687ca54e1a05b1e5e12eae956b65c3a"} Nov 01 04:30:03.240380 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:03.240305 6735 scope.go:122] "RemoveContainer" containerID="0ef953ea366de869e61ef2646bf04e660fedc5d3fcd9d372cbc1b58982e8cc2f" Nov 01 04:30:03.240583 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:03.240566 6735 scope.go:122] "RemoveContainer" containerID="d8815523cbebd9e0c26ffb9350086ec65687ca54e1a05b1e5e12eae956b65c3a" Nov 01 04:30:03.240832 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:03.240809 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk_odh-ai-gateway-infra(401bdad5-fe06-4bf6-b145-5ae0a9af9e90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" Nov 01 04:30:03.344042 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:03.343992 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" podStartSLOduration=3.343979204 podStartE2EDuration="3.343979204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:30:03.342187385 +0000 UTC m=+2814.931282036" watchObservedRunningTime="2026-11-01 04:30:03.343979204 +0000 UTC m=+2814.933073833" Nov 01 04:30:04.247379 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:04.247341 6735 generic.go:356] "Generic (PLEG): container finished" podID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerID="bbd8428ffb40157a54f499f070cf38ef751317654287dfb03bd8b6ffb8eeb764" exitCode=60 Nov 01 04:30:04.247834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:04.247418 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" event={"ID":"1e44b907-7cec-40a8-87f4-3f697a247d25","Type":"ContainerDied","Data":"bbd8428ffb40157a54f499f070cf38ef751317654287dfb03bd8b6ffb8eeb764"} Nov 01 04:30:04.247834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:04.247478 6735 scope.go:122] "RemoveContainer" containerID="eb8c3bbc8da37f12c10d76b3dcc9708acaba77ea22e39c4c900e7fe84bef9fa1" Nov 01 04:30:04.247834 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:04.247709 6735 scope.go:122] "RemoveContainer" containerID="bbd8428ffb40157a54f499f070cf38ef751317654287dfb03bd8b6ffb8eeb764" Nov 01 04:30:04.248027 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:04.248006 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8_odh-ai-gateway-infra(1e44b907-7cec-40a8-87f4-3f697a247d25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" Nov 01 04:30:04.249678 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:04.249662 6735 scope.go:122] "RemoveContainer" containerID="d8815523cbebd9e0c26ffb9350086ec65687ca54e1a05b1e5e12eae956b65c3a" Nov 01 04:30:04.249894 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:04.249877 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk_odh-ai-gateway-infra(401bdad5-fe06-4bf6-b145-5ae0a9af9e90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" Nov 01 04:30:05.256683 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.256655 6735 scope.go:122] "RemoveContainer" containerID="bbd8428ffb40157a54f499f070cf38ef751317654287dfb03bd8b6ffb8eeb764" Nov 01 04:30:05.257077 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:05.256902 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8_odh-ai-gateway-infra(1e44b907-7cec-40a8-87f4-3f697a247d25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" Nov 01 04:30:05.375774 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.375741 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv"] Nov 01 04:30:05.379139 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.379116 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.427522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.427492 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 04:30:05.435344 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.435315 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv"] Nov 01 04:30:05.495981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.495939 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.496179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.496024 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8klx6\" (UniqueName: \"kubernetes.io/projected/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-kube-api-access-8klx6\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.496179 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.496129 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.597553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.597475 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.597553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.597517 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8klx6\" (UniqueName: \"kubernetes.io/projected/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-kube-api-access-8klx6\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.597752 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.597589 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.598295 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.598269 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.600090 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.600062 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.655593 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.655559 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8klx6\" (UniqueName: \"kubernetes.io/projected/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-kube-api-access-8klx6\") pod \"maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:05.691492 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:05.691462 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:06.054769 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:06.054740 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv"] Nov 01 04:30:06.057197 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:30:06.057157 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda983147c_7cfb_4f6c_8c8f_6ff708f1cbe6.slice/crio-8de8a173c4572c02d2a6cb44158bfb85f5f2972453c3e0f4df9dccb1cc92222c WatchSource:0}: Error finding container 8de8a173c4572c02d2a6cb44158bfb85f5f2972453c3e0f4df9dccb1cc92222c: Status 404 returned error can't find the container with id 8de8a173c4572c02d2a6cb44158bfb85f5f2972453c3e0f4df9dccb1cc92222c Nov 01 04:30:06.263474 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:06.263439 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" event={"ID":"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6","Type":"ContainerStarted","Data":"8de8a173c4572c02d2a6cb44158bfb85f5f2972453c3e0f4df9dccb1cc92222c"} Nov 01 04:30:07.270315 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:07.270276 6735 generic.go:356] "Generic (PLEG): container finished" podID="a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" containerID="31e69848c25e25c058ab6b5dc33fbc8a4bdf9cec10a320e6989899d8451e58eb" exitCode=0 Nov 01 04:30:07.270728 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:07.270370 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" event={"ID":"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6","Type":"ContainerDied","Data":"31e69848c25e25c058ab6b5dc33fbc8a4bdf9cec10a320e6989899d8451e58eb"} Nov 01 04:30:08.420027 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.420000 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:08.528110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.528016 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-xks-ca\") pod \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " Nov 01 04:30:08.528110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.528100 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-openshift-ca\") pod \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " Nov 01 04:30:08.528336 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.528132 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-kube-api-access-8klx6\" (UniqueName: \"kubernetes.io/projected/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-kube-api-access-8klx6\") pod \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\" (UID: \"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6\") " Nov 01 04:30:08.528774 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.528710 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-openshift-ca" pod "a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" (UID: "a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:30:08.530704 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.530666 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-kube-api-access-8klx6" pod "a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" (UID: "a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6"). InnerVolumeSpecName "kube-api-access-8klx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:30:08.531206 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.531177 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-xks-ca" pod "a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" (UID: "a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:30:08.629052 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.629010 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-xks-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:08.629052 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.629050 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-maas-api-openshift-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:08.629254 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:08.629066 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8klx6\" (UniqueName: \"kubernetes.io/projected/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6-kube-api-access-8klx6\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:09.281445 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:09.281415 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" Nov 01 04:30:09.281633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:09.281445 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv" event={"ID":"a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6","Type":"ContainerDied","Data":"8de8a173c4572c02d2a6cb44158bfb85f5f2972453c3e0f4df9dccb1cc92222c"} Nov 01 04:30:09.281633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:09.281486 6735 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de8a173c4572c02d2a6cb44158bfb85f5f2972453c3e0f4df9dccb1cc92222c" Nov 01 04:30:13.824496 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:13.824463 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587"] Nov 01 04:30:13.925347 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:13.925313 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p"] Nov 01 04:30:14.518378 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:14.518346 6735 scope.go:122] "RemoveContainer" containerID="d8815523cbebd9e0c26ffb9350086ec65687ca54e1a05b1e5e12eae956b65c3a" Nov 01 04:30:15.319370 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:15.319337 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" event={"ID":"401bdad5-fe06-4bf6-b145-5ae0a9af9e90","Type":"ContainerStarted","Data":"95d432cf8462de20f6206cb266f1a0d7e2d376f0141b7341b4a053e959121578"} Nov 01 04:30:16.289013 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:16.288976 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv"] Nov 01 04:30:16.326633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:16.326604 6735 generic.go:356] "Generic (PLEG): container finished" podID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerID="95d432cf8462de20f6206cb266f1a0d7e2d376f0141b7341b4a053e959121578" exitCode=60 Nov 01 04:30:16.326633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:16.326637 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" event={"ID":"401bdad5-fe06-4bf6-b145-5ae0a9af9e90","Type":"ContainerDied","Data":"95d432cf8462de20f6206cb266f1a0d7e2d376f0141b7341b4a053e959121578"} Nov 01 04:30:16.327120 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:16.326682 6735 scope.go:122] "RemoveContainer" containerID="d8815523cbebd9e0c26ffb9350086ec65687ca54e1a05b1e5e12eae956b65c3a" Nov 01 04:30:16.327120 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:16.326978 6735 scope.go:122] "RemoveContainer" containerID="95d432cf8462de20f6206cb266f1a0d7e2d376f0141b7341b4a053e959121578" Nov 01 04:30:16.327280 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:16.327256 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk_odh-ai-gateway-infra(401bdad5-fe06-4bf6-b145-5ae0a9af9e90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" Nov 01 04:30:16.436801 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:16.436765 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk"] Nov 01 04:30:16.518821 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:16.518791 6735 scope.go:122] "RemoveContainer" containerID="bbd8428ffb40157a54f499f070cf38ef751317654287dfb03bd8b6ffb8eeb764" Nov 01 04:30:17.333028 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.332996 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" event={"ID":"1e44b907-7cec-40a8-87f4-3f697a247d25","Type":"ContainerStarted","Data":"abc4cd0c9c56cee5a987b85f08876813fbd77610fe82569111bea53545a82507"} Nov 01 04:30:17.466327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.466300 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:17.516450 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.516420 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-kube-api-access-jrhrd\" (UniqueName: \"kubernetes.io/projected/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-kube-api-access-jrhrd\") pod \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " Nov 01 04:30:17.516605 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.516542 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-maas-api-service-ca\") pod \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\" (UID: \"401bdad5-fe06-4bf6-b145-5ae0a9af9e90\") " Nov 01 04:30:17.516944 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.516922 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-maas-api-service-ca" pod "401bdad5-fe06-4bf6-b145-5ae0a9af9e90" (UID: "401bdad5-fe06-4bf6-b145-5ae0a9af9e90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:30:17.518835 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.518812 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-kube-api-access-jrhrd" pod "401bdad5-fe06-4bf6-b145-5ae0a9af9e90" (UID: "401bdad5-fe06-4bf6-b145-5ae0a9af9e90"). InnerVolumeSpecName "kube-api-access-jrhrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:30:17.618095 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.618013 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-maas-api-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:17.618095 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:17.618061 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhrd\" (UniqueName: \"kubernetes.io/projected/401bdad5-fe06-4bf6-b145-5ae0a9af9e90-kube-api-access-jrhrd\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:18.340825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.340792 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" Nov 01 04:30:18.340825 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.340813 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk" event={"ID":"401bdad5-fe06-4bf6-b145-5ae0a9af9e90","Type":"ContainerDied","Data":"50979c5ff25e8feb94b62867704845e97ef1bcaa2dc6e016b48827b925174a0f"} Nov 01 04:30:18.341414 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.340869 6735 scope.go:122] "RemoveContainer" containerID="95d432cf8462de20f6206cb266f1a0d7e2d376f0141b7341b4a053e959121578" Nov 01 04:30:18.343256 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.343233 6735 generic.go:356] "Generic (PLEG): container finished" podID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerID="abc4cd0c9c56cee5a987b85f08876813fbd77610fe82569111bea53545a82507" exitCode=60 Nov 01 04:30:18.343353 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.343288 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" event={"ID":"1e44b907-7cec-40a8-87f4-3f697a247d25","Type":"ContainerDied","Data":"abc4cd0c9c56cee5a987b85f08876813fbd77610fe82569111bea53545a82507"} Nov 01 04:30:18.343603 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.343590 6735 scope.go:122] "RemoveContainer" containerID="abc4cd0c9c56cee5a987b85f08876813fbd77610fe82569111bea53545a82507" Nov 01 04:30:18.343804 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:18.343788 6735 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8_odh-ai-gateway-infra(1e44b907-7cec-40a8-87f4-3f697a247d25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" Nov 01 04:30:18.350150 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.350135 6735 scope.go:122] "RemoveContainer" containerID="bbd8428ffb40157a54f499f070cf38ef751317654287dfb03bd8b6ffb8eeb764" Nov 01 04:30:18.451121 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.451086 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8"] Nov 01 04:30:18.602395 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.602320 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk"] Nov 01 04:30:18.605687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.605657 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-865066-29891790-hsbmk"] Nov 01 04:30:18.825199 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.825157 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" podUID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" containerName="payload-processing" containerID="cri-o://b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb" gracePeriod=25 Nov 01 04:30:18.926157 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:18.926072 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" podUID="01ecd83a-560d-4e93-81cb-3474b8d9881b" containerName="payload-pre-processing" containerID="cri-o://0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797" gracePeriod=25 Nov 01 04:30:19.480317 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:19.480294 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:19.539079 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:19.539011 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e44b907-7cec-40a8-87f4-3f697a247d25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44b907-7cec-40a8-87f4-3f697a247d25-maas-api-service-ca\") pod \"1e44b907-7cec-40a8-87f4-3f697a247d25\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " Nov 01 04:30:19.539235 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:19.539121 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e44b907-7cec-40a8-87f4-3f697a247d25-kube-api-access-5xcgj\" (UniqueName: \"kubernetes.io/projected/1e44b907-7cec-40a8-87f4-3f697a247d25-kube-api-access-5xcgj\") pod \"1e44b907-7cec-40a8-87f4-3f697a247d25\" (UID: \"1e44b907-7cec-40a8-87f4-3f697a247d25\") " Nov 01 04:30:19.539401 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:19.539381 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e44b907-7cec-40a8-87f4-3f697a247d25-maas-api-service-ca" pod "1e44b907-7cec-40a8-87f4-3f697a247d25" (UID: "1e44b907-7cec-40a8-87f4-3f697a247d25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:30:19.540019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:19.539946 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e44b907-7cec-40a8-87f4-3f697a247d25-maas-api-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:19.541664 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:19.541644 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e44b907-7cec-40a8-87f4-3f697a247d25-kube-api-access-5xcgj" pod "1e44b907-7cec-40a8-87f4-3f697a247d25" (UID: "1e44b907-7cec-40a8-87f4-3f697a247d25"). InnerVolumeSpecName "kube-api-access-5xcgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:30:19.641261 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:19.641184 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xcgj\" (UniqueName: \"kubernetes.io/projected/1e44b907-7cec-40a8-87f4-3f697a247d25-kube-api-access-5xcgj\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:20.365387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:20.365350 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" event={"ID":"1e44b907-7cec-40a8-87f4-3f697a247d25","Type":"ContainerDied","Data":"51a561068e99bb8c5c9b2f62182bb0227a9cdc0ac07be0fa9d1c0df2e484e916"} Nov 01 04:30:20.365387 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:20.365377 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8" Nov 01 04:30:20.365643 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:20.365448 6735 scope.go:122] "RemoveContainer" containerID="abc4cd0c9c56cee5a987b85f08876813fbd77610fe82569111bea53545a82507" Nov 01 04:30:20.525279 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:20.525249 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" path="/var/lib/kubelet/pods/401bdad5-fe06-4bf6-b145-5ae0a9af9e90/volumes" Nov 01 04:30:20.525660 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:20.525555 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8"] Nov 01 04:30:20.565511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:20.565483 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-b8319b-29891790-8kfw8"] Nov 01 04:30:21.289655 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:21.289613 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" podUID="78b560bf-0e29-416b-9f70-00223c46a8d1" containerName="payload-pre-processing" containerID="cri-o://d222fd530c181797415f7f43122fb9e13027201c7c8bd9c5ba19f1dce90f4a30" gracePeriod=25 Nov 01 04:30:22.526071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:22.526011 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" path="/var/lib/kubelet/pods/1e44b907-7cec-40a8-87f4-3f697a247d25/volumes" Nov 01 04:30:24.802772 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:24.802728 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" podUID="78b560bf-0e29-416b-9f70-00223c46a8d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:30:26.239398 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:26.239363 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" podUID="01ecd83a-560d-4e93-81cb-3474b8d9881b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:30:26.239857 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:26.239361 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" podUID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:30:30.467089 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:30.467055 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:30:33.001091 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:33.001056 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:30:34.801943 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:34.801901 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" podUID="78b560bf-0e29-416b-9f70-00223c46a8d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:30:36.239003 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:36.238967 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" podUID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:30:36.239412 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:36.239012 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" podUID="01ecd83a-560d-4e93-81cb-3474b8d9881b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:30:37.365829 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.365797 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb"] Nov 01 04:30:37.366272 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.366005 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" podUID="41a190b3-d99e-46f1-841c-9aa77bda2ec5" containerName="maas-api" containerID="cri-o://993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817" gracePeriod=30 Nov 01 04:30:37.640665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.640641 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:30:37.719082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.719049 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a190b3-d99e-46f1-841c-9aa77bda2ec5-kube-api-access-ltvvk\" (UniqueName: \"kubernetes.io/projected/41a190b3-d99e-46f1-841c-9aa77bda2ec5-kube-api-access-ltvvk\") pod \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " Nov 01 04:30:37.719248 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.719116 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a190b3-d99e-46f1-841c-9aa77bda2ec5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a190b3-d99e-46f1-841c-9aa77bda2ec5-openshift-service-ca\") pod \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " Nov 01 04:30:37.719287 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.719244 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-metrics-certs\") pod \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " Nov 01 04:30:37.719339 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.719321 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls\") pod \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\" (UID: \"41a190b3-d99e-46f1-841c-9aa77bda2ec5\") " Nov 01 04:30:37.719529 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.719504 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a190b3-d99e-46f1-841c-9aa77bda2ec5-openshift-service-ca" pod "41a190b3-d99e-46f1-841c-9aa77bda2ec5" (UID: "41a190b3-d99e-46f1-841c-9aa77bda2ec5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:30:37.719677 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.719658 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a190b3-d99e-46f1-841c-9aa77bda2ec5-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:37.721460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.721426 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-metrics-certs" pod "41a190b3-d99e-46f1-841c-9aa77bda2ec5" (UID: "41a190b3-d99e-46f1-841c-9aa77bda2ec5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:30:37.721460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.721440 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a190b3-d99e-46f1-841c-9aa77bda2ec5-kube-api-access-ltvvk" pod "41a190b3-d99e-46f1-841c-9aa77bda2ec5" (UID: "41a190b3-d99e-46f1-841c-9aa77bda2ec5"). InnerVolumeSpecName "kube-api-access-ltvvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:30:37.721460 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.721449 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls" pod "41a190b3-d99e-46f1-841c-9aa77bda2ec5" (UID: "41a190b3-d99e-46f1-841c-9aa77bda2ec5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:30:37.820477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.820438 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:37.820477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.820470 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a190b3-d99e-46f1-841c-9aa77bda2ec5-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:37.820477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:37.820481 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvvk\" (UniqueName: \"kubernetes.io/projected/41a190b3-d99e-46f1-841c-9aa77bda2ec5-kube-api-access-ltvvk\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:38.463502 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.463471 6735 generic.go:356] "Generic (PLEG): container finished" podID="41a190b3-d99e-46f1-841c-9aa77bda2ec5" containerID="993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817" exitCode=0 Nov 01 04:30:38.464008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.463522 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" Nov 01 04:30:38.464008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.463551 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" event={"ID":"41a190b3-d99e-46f1-841c-9aa77bda2ec5","Type":"ContainerDied","Data":"993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817"} Nov 01 04:30:38.464008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.463604 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb" event={"ID":"41a190b3-d99e-46f1-841c-9aa77bda2ec5","Type":"ContainerDied","Data":"040680758c068344d7af940f361fb7b7d3ddafa4128e7d2c61d5b3e83d16e0d8"} Nov 01 04:30:38.464008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.463630 6735 scope.go:122] "RemoveContainer" containerID="993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817" Nov 01 04:30:38.473511 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.473490 6735 scope.go:122] "RemoveContainer" containerID="993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817" Nov 01 04:30:38.473790 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:38.473773 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817\": container with ID starting with 993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817 not found: ID does not exist" containerID="993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817" Nov 01 04:30:38.473790 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.473797 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817"} err="failed to get container status \"993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817\": rpc error: code = NotFound desc = could not find container \"993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817\": container with ID starting with 993b847206f7e61bf979397963da890512d033e0511b5ea20a4134aea9163817 not found: ID does not exist" Nov 01 04:30:38.616938 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.616903 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb"] Nov 01 04:30:38.620788 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:38.620758 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b8319b-7b46f4d45b-dqhnb"] Nov 01 04:30:39.089343 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.089281 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:30:39.185573 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.185546 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:30:39.236231 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.236201 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97aa7198-e7c4-43f4-a101-f80f82bf3f70-kube-api-access-pcwtq\" (UniqueName: \"kubernetes.io/projected/97aa7198-e7c4-43f4-a101-f80f82bf3f70-kube-api-access-pcwtq\") pod \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " Nov 01 04:30:39.236407 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.236262 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97aa7198-e7c4-43f4-a101-f80f82bf3f70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aa7198-e7c4-43f4-a101-f80f82bf3f70-plugins-config-volume\") pod \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\" (UID: \"97aa7198-e7c4-43f4-a101-f80f82bf3f70\") " Nov 01 04:30:39.236656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.236633 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97aa7198-e7c4-43f4-a101-f80f82bf3f70-plugins-config-volume" pod "97aa7198-e7c4-43f4-a101-f80f82bf3f70" (UID: "97aa7198-e7c4-43f4-a101-f80f82bf3f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:30:39.238382 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.238364 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aa7198-e7c4-43f4-a101-f80f82bf3f70-kube-api-access-pcwtq" pod "97aa7198-e7c4-43f4-a101-f80f82bf3f70" (UID: "97aa7198-e7c4-43f4-a101-f80f82bf3f70"). InnerVolumeSpecName "kube-api-access-pcwtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:30:39.337503 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.337475 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ecd83a-560d-4e93-81cb-3474b8d9881b-kube-api-access-zlgr2\" (UniqueName: \"kubernetes.io/projected/01ecd83a-560d-4e93-81cb-3474b8d9881b-kube-api-access-zlgr2\") pod \"01ecd83a-560d-4e93-81cb-3474b8d9881b\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " Nov 01 04:30:39.337701 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.337527 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ecd83a-560d-4e93-81cb-3474b8d9881b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ecd83a-560d-4e93-81cb-3474b8d9881b-plugins-config-volume\") pod \"01ecd83a-560d-4e93-81cb-3474b8d9881b\" (UID: \"01ecd83a-560d-4e93-81cb-3474b8d9881b\") " Nov 01 04:30:39.337781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.337712 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aa7198-e7c4-43f4-a101-f80f82bf3f70-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:39.337781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.337722 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcwtq\" (UniqueName: \"kubernetes.io/projected/97aa7198-e7c4-43f4-a101-f80f82bf3f70-kube-api-access-pcwtq\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:39.337973 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.337950 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ecd83a-560d-4e93-81cb-3474b8d9881b-plugins-config-volume" pod "01ecd83a-560d-4e93-81cb-3474b8d9881b" (UID: "01ecd83a-560d-4e93-81cb-3474b8d9881b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:30:39.339687 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.339643 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ecd83a-560d-4e93-81cb-3474b8d9881b-kube-api-access-zlgr2" pod "01ecd83a-560d-4e93-81cb-3474b8d9881b" (UID: "01ecd83a-560d-4e93-81cb-3474b8d9881b"). InnerVolumeSpecName "kube-api-access-zlgr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:30:39.439004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.438973 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgr2\" (UniqueName: \"kubernetes.io/projected/01ecd83a-560d-4e93-81cb-3474b8d9881b-kube-api-access-zlgr2\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:39.439004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.438997 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ecd83a-560d-4e93-81cb-3474b8d9881b-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:39.470226 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.470197 6735 generic.go:356] "Generic (PLEG): container finished" podID="01ecd83a-560d-4e93-81cb-3474b8d9881b" containerID="0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797" exitCode=0 Nov 01 04:30:39.470656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.470248 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" Nov 01 04:30:39.470656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.470285 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" event={"ID":"01ecd83a-560d-4e93-81cb-3474b8d9881b","Type":"ContainerDied","Data":"0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797"} Nov 01 04:30:39.470656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.470341 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p" event={"ID":"01ecd83a-560d-4e93-81cb-3474b8d9881b","Type":"ContainerDied","Data":"7b53e815a4e401d86987e8020d4824c2161dfbd7797133b09a5691ec8812e58b"} Nov 01 04:30:39.470656 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.470372 6735 scope.go:122] "RemoveContainer" containerID="0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797" Nov 01 04:30:39.471714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.471607 6735 generic.go:356] "Generic (PLEG): container finished" podID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" containerID="b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb" exitCode=0 Nov 01 04:30:39.471714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.471644 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" Nov 01 04:30:39.471714 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.471676 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" event={"ID":"97aa7198-e7c4-43f4-a101-f80f82bf3f70","Type":"ContainerDied","Data":"b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb"} Nov 01 04:30:39.471850 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.471717 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587" event={"ID":"97aa7198-e7c4-43f4-a101-f80f82bf3f70","Type":"ContainerDied","Data":"ca85e5b0a5f90bffe3b40c596c0ba07a78ae3bddd894b1fa75fe6b5cba52fa08"} Nov 01 04:30:39.480870 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.480852 6735 scope.go:122] "RemoveContainer" containerID="0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797" Nov 01 04:30:39.481219 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:39.481187 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797\": container with ID starting with 0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797 not found: ID does not exist" containerID="0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797" Nov 01 04:30:39.481328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.481222 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797"} err="failed to get container status \"0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797\": rpc error: code = NotFound desc = could not find container \"0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797\": container with ID starting with 0070647a605025fcb6685d1e0b927a5b819976f5c8fb7d1fc0e6dccba2114797 not found: ID does not exist" Nov 01 04:30:39.481328 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.481244 6735 scope.go:122] "RemoveContainer" containerID="b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb" Nov 01 04:30:39.489747 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.489725 6735 scope.go:122] "RemoveContainer" containerID="b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb" Nov 01 04:30:39.490029 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:30:39.490011 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb\": container with ID starting with b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb not found: ID does not exist" containerID="b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb" Nov 01 04:30:39.490128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.490064 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb"} err="failed to get container status \"b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb\": rpc error: code = NotFound desc = could not find container \"b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb\": container with ID starting with b5219957f8a49a12910ed1d41d74a04a99d1a193abfb7c937b40a73de8364ceb not found: ID does not exist" Nov 01 04:30:39.632614 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.632540 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587"] Nov 01 04:30:39.637368 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.637346 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ad9e22-f8c6dbd69-ts587"] Nov 01 04:30:39.738818 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.738787 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p"] Nov 01 04:30:39.744863 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:39.744837 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ad9e22-7bd5748db7-ltg8p"] Nov 01 04:30:40.524364 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:40.524332 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ecd83a-560d-4e93-81cb-3474b8d9881b" path="/var/lib/kubelet/pods/01ecd83a-560d-4e93-81cb-3474b8d9881b/volumes" Nov 01 04:30:40.524745 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:40.524737 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a190b3-d99e-46f1-841c-9aa77bda2ec5" path="/var/lib/kubelet/pods/41a190b3-d99e-46f1-841c-9aa77bda2ec5/volumes" Nov 01 04:30:40.525143 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:40.525127 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" path="/var/lib/kubelet/pods/97aa7198-e7c4-43f4-a101-f80f82bf3f70/volumes" Nov 01 04:30:41.495630 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.495575 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" event={"ID":"78b560bf-0e29-416b-9f70-00223c46a8d1","Type":"ContainerDied","Data":"d222fd530c181797415f7f43122fb9e13027201c7c8bd9c5ba19f1dce90f4a30"} Nov 01 04:30:41.495832 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.495708 6735 generic.go:356] "Generic (PLEG): container finished" podID="78b560bf-0e29-416b-9f70-00223c46a8d1" containerID="d222fd530c181797415f7f43122fb9e13027201c7c8bd9c5ba19f1dce90f4a30" exitCode=0 Nov 01 04:30:41.561380 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.561313 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 04:30:41.661314 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.661279 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b560bf-0e29-416b-9f70-00223c46a8d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b560bf-0e29-416b-9f70-00223c46a8d1-plugins-config-volume\") pod \"78b560bf-0e29-416b-9f70-00223c46a8d1\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " Nov 01 04:30:41.661499 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.661342 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b560bf-0e29-416b-9f70-00223c46a8d1-kube-api-access-rgfs2\" (UniqueName: \"kubernetes.io/projected/78b560bf-0e29-416b-9f70-00223c46a8d1-kube-api-access-rgfs2\") pod \"78b560bf-0e29-416b-9f70-00223c46a8d1\" (UID: \"78b560bf-0e29-416b-9f70-00223c46a8d1\") " Nov 01 04:30:41.661646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.661623 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b560bf-0e29-416b-9f70-00223c46a8d1-plugins-config-volume" pod "78b560bf-0e29-416b-9f70-00223c46a8d1" (UID: "78b560bf-0e29-416b-9f70-00223c46a8d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:30:41.661747 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.661732 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78b560bf-0e29-416b-9f70-00223c46a8d1-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:41.663566 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.663541 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b560bf-0e29-416b-9f70-00223c46a8d1-kube-api-access-rgfs2" pod "78b560bf-0e29-416b-9f70-00223c46a8d1" (UID: "78b560bf-0e29-416b-9f70-00223c46a8d1"). InnerVolumeSpecName "kube-api-access-rgfs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:30:41.763107 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:41.763066 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgfs2\" (UniqueName: \"kubernetes.io/projected/78b560bf-0e29-416b-9f70-00223c46a8d1-kube-api-access-rgfs2\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:30:42.501698 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:42.501671 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" Nov 01 04:30:42.501888 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:42.501710 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv" event={"ID":"78b560bf-0e29-416b-9f70-00223c46a8d1","Type":"ContainerDied","Data":"d5dca37892ef4c0515e928fadbdff867ba68f6c9d2ed029256c8deb774bf2d03"} Nov 01 04:30:42.501888 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:42.501769 6735 scope.go:122] "RemoveContainer" containerID="d222fd530c181797415f7f43122fb9e13027201c7c8bd9c5ba19f1dce90f4a30" Nov 01 04:30:42.663391 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:42.663358 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv"] Nov 01 04:30:42.709685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:42.709655 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2c259-564f6fc64c-vgfzv"] Nov 01 04:30:44.524527 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:44.524490 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b560bf-0e29-416b-9f70-00223c46a8d1" path="/var/lib/kubelet/pods/78b560bf-0e29-416b-9f70-00223c46a8d1/volumes" Nov 01 04:30:51.649162 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:51.649128 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc"] Nov 01 04:30:51.694539 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:51.694505 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9"] Nov 01 04:30:57.517896 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:57.517855 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:30:58.684088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:58.684058 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg"] Nov 01 04:30:58.782074 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:30:58.782018 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq"] Nov 01 04:31:03.000830 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:31:03.000792 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:31:03.684635 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:03.684591 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" podUID="73b88c83-a504-44d4-8748-6ecb3200bbe1" containerName="payload-processing" containerID="cri-o://67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18" gracePeriod=25 Nov 01 04:31:03.782443 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:03.782395 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" podUID="4ca23905-14e8-4101-903b-bf0631968c1f" containerName="payload-pre-processing" containerID="cri-o://24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce" gracePeriod=25 Nov 01 04:31:03.937221 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:03.937159 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 04:31:03.983759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:03.983730 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b88c83-a504-44d4-8748-6ecb3200bbe1-kube-api-access-r8nrq\" (UniqueName: \"kubernetes.io/projected/73b88c83-a504-44d4-8748-6ecb3200bbe1-kube-api-access-r8nrq\") pod \"73b88c83-a504-44d4-8748-6ecb3200bbe1\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " Nov 01 04:31:03.983927 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:03.983781 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b88c83-a504-44d4-8748-6ecb3200bbe1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b88c83-a504-44d4-8748-6ecb3200bbe1-plugins-config-volume\") pod \"73b88c83-a504-44d4-8748-6ecb3200bbe1\" (UID: \"73b88c83-a504-44d4-8748-6ecb3200bbe1\") " Nov 01 04:31:03.984311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:03.984283 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b88c83-a504-44d4-8748-6ecb3200bbe1-plugins-config-volume" pod "73b88c83-a504-44d4-8748-6ecb3200bbe1" (UID: "73b88c83-a504-44d4-8748-6ecb3200bbe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:31:03.986064 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:03.986025 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b88c83-a504-44d4-8748-6ecb3200bbe1-kube-api-access-r8nrq" pod "73b88c83-a504-44d4-8748-6ecb3200bbe1" (UID: "73b88c83-a504-44d4-8748-6ecb3200bbe1"). InnerVolumeSpecName "kube-api-access-r8nrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:31:04.038547 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.038522 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 04:31:04.085186 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.085145 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca23905-14e8-4101-903b-bf0631968c1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca23905-14e8-4101-903b-bf0631968c1f-plugins-config-volume\") pod \"4ca23905-14e8-4101-903b-bf0631968c1f\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " Nov 01 04:31:04.085374 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.085296 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca23905-14e8-4101-903b-bf0631968c1f-kube-api-access-jn65l\" (UniqueName: \"kubernetes.io/projected/4ca23905-14e8-4101-903b-bf0631968c1f-kube-api-access-jn65l\") pod \"4ca23905-14e8-4101-903b-bf0631968c1f\" (UID: \"4ca23905-14e8-4101-903b-bf0631968c1f\") " Nov 01 04:31:04.085540 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.085516 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca23905-14e8-4101-903b-bf0631968c1f-plugins-config-volume" pod "4ca23905-14e8-4101-903b-bf0631968c1f" (UID: "4ca23905-14e8-4101-903b-bf0631968c1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:31:04.085628 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.085614 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8nrq\" (UniqueName: \"kubernetes.io/projected/73b88c83-a504-44d4-8748-6ecb3200bbe1-kube-api-access-r8nrq\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:04.085665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.085631 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b88c83-a504-44d4-8748-6ecb3200bbe1-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:04.085665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.085641 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca23905-14e8-4101-903b-bf0631968c1f-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:04.087539 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.087513 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca23905-14e8-4101-903b-bf0631968c1f-kube-api-access-jn65l" pod "4ca23905-14e8-4101-903b-bf0631968c1f" (UID: "4ca23905-14e8-4101-903b-bf0631968c1f"). InnerVolumeSpecName "kube-api-access-jn65l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:31:04.186405 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.186370 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn65l\" (UniqueName: \"kubernetes.io/projected/4ca23905-14e8-4101-903b-bf0631968c1f-kube-api-access-jn65l\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:04.484657 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.484622 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn"] Nov 01 04:31:04.484845 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.484817 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" podUID="74c5ccae-f7ae-41be-bca6-decf8a9d8481" containerName="maas-api" containerID="cri-o://9921c6abaedc1fbb7e1be9470e9d5a8b51ac0cd265908d202733b41316e10f54" gracePeriod=30 Nov 01 04:31:04.616986 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.616954 6735 generic.go:356] "Generic (PLEG): container finished" podID="74c5ccae-f7ae-41be-bca6-decf8a9d8481" containerID="9921c6abaedc1fbb7e1be9470e9d5a8b51ac0cd265908d202733b41316e10f54" exitCode=0 Nov 01 04:31:04.617175 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.617048 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" event={"ID":"74c5ccae-f7ae-41be-bca6-decf8a9d8481","Type":"ContainerDied","Data":"9921c6abaedc1fbb7e1be9470e9d5a8b51ac0cd265908d202733b41316e10f54"} Nov 01 04:31:04.618521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.618491 6735 generic.go:356] "Generic (PLEG): container finished" podID="4ca23905-14e8-4101-903b-bf0631968c1f" containerID="24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce" exitCode=0 Nov 01 04:31:04.618521 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.618524 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" event={"ID":"4ca23905-14e8-4101-903b-bf0631968c1f","Type":"ContainerDied","Data":"24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce"} Nov 01 04:31:04.618734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.618560 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" Nov 01 04:31:04.618734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.618574 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq" event={"ID":"4ca23905-14e8-4101-903b-bf0631968c1f","Type":"ContainerDied","Data":"7ce3067a74c84e3ff27856c8c5bd99cc9fb2d3ff5bfc1e8226de3f7e143fb11b"} Nov 01 04:31:04.618734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.618605 6735 scope.go:122] "RemoveContainer" containerID="24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce" Nov 01 04:31:04.620346 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.620326 6735 generic.go:356] "Generic (PLEG): container finished" podID="73b88c83-a504-44d4-8748-6ecb3200bbe1" containerID="67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18" exitCode=0 Nov 01 04:31:04.620476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.620379 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" Nov 01 04:31:04.620476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.620432 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" event={"ID":"73b88c83-a504-44d4-8748-6ecb3200bbe1","Type":"ContainerDied","Data":"67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18"} Nov 01 04:31:04.620476 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.620473 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg" event={"ID":"73b88c83-a504-44d4-8748-6ecb3200bbe1","Type":"ContainerDied","Data":"9f0c507dfcab841edd8b963c2d4aa54eeddd4c434b54339c28eab56eea8d09b4"} Nov 01 04:31:04.628298 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.628272 6735 scope.go:122] "RemoveContainer" containerID="24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce" Nov 01 04:31:04.628681 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:31:04.628658 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce\": container with ID starting with 24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce not found: ID does not exist" containerID="24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce" Nov 01 04:31:04.628764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.628694 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce"} err="failed to get container status \"24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce\": rpc error: code = NotFound desc = could not find container \"24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce\": container with ID starting with 24f1c3142cb1d91a183097268b691a963bf48bdc9055845488d91a497e3488ce not found: ID does not exist" Nov 01 04:31:04.628764 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.628720 6735 scope.go:122] "RemoveContainer" containerID="67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18" Nov 01 04:31:04.638805 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.638785 6735 scope.go:122] "RemoveContainer" containerID="67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18" Nov 01 04:31:04.639120 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:31:04.639099 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18\": container with ID starting with 67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18 not found: ID does not exist" containerID="67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18" Nov 01 04:31:04.639186 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.639128 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18"} err="failed to get container status \"67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18\": rpc error: code = NotFound desc = could not find container \"67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18\": container with ID starting with 67f7cad0e99d24fc96198e95c13d050d36bd60dfd8ebffc078b09d099fb26c18 not found: ID does not exist" Nov 01 04:31:04.773907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.773878 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 04:31:04.774722 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.774703 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg"] Nov 01 04:31:04.789973 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.782768 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a9c5b3-5bd4ff54f-dsmwg"] Nov 01 04:31:04.887876 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.887842 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq"] Nov 01 04:31:04.893637 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.893604 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a9c5b3-58fbb5d6bc-8hdkq"] Nov 01 04:31:04.893847 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.893827 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c5ccae-f7ae-41be-bca6-decf8a9d8481-kube-api-access-b9jhw\" (UniqueName: \"kubernetes.io/projected/74c5ccae-f7ae-41be-bca6-decf8a9d8481-kube-api-access-b9jhw\") pod \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " Nov 01 04:31:04.893936 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.893923 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls\") pod \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " Nov 01 04:31:04.894004 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.893993 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-metrics-certs\") pod \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " Nov 01 04:31:04.894071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.894026 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c5ccae-f7ae-41be-bca6-decf8a9d8481-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c5ccae-f7ae-41be-bca6-decf8a9d8481-openshift-service-ca\") pod \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\" (UID: \"74c5ccae-f7ae-41be-bca6-decf8a9d8481\") " Nov 01 04:31:04.894427 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.894403 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c5ccae-f7ae-41be-bca6-decf8a9d8481-openshift-service-ca" pod "74c5ccae-f7ae-41be-bca6-decf8a9d8481" (UID: "74c5ccae-f7ae-41be-bca6-decf8a9d8481"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:31:04.896274 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.896244 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-metrics-certs" pod "74c5ccae-f7ae-41be-bca6-decf8a9d8481" (UID: "74c5ccae-f7ae-41be-bca6-decf8a9d8481"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:31:04.896369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.896277 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls" pod "74c5ccae-f7ae-41be-bca6-decf8a9d8481" (UID: "74c5ccae-f7ae-41be-bca6-decf8a9d8481"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:31:04.896369 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.896295 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c5ccae-f7ae-41be-bca6-decf8a9d8481-kube-api-access-b9jhw" pod "74c5ccae-f7ae-41be-bca6-decf8a9d8481" (UID: "74c5ccae-f7ae-41be-bca6-decf8a9d8481"). InnerVolumeSpecName "kube-api-access-b9jhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:31:04.995759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.995722 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:04.995759 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.995763 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c5ccae-f7ae-41be-bca6-decf8a9d8481-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:04.995963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.995779 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jhw\" (UniqueName: \"kubernetes.io/projected/74c5ccae-f7ae-41be-bca6-decf8a9d8481-kube-api-access-b9jhw\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:04.995963 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:04.995795 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74c5ccae-f7ae-41be-bca6-decf8a9d8481-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:05.626907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:05.626873 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" Nov 01 04:31:05.626907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:05.626898 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn" event={"ID":"74c5ccae-f7ae-41be-bca6-decf8a9d8481","Type":"ContainerDied","Data":"f36089bbdf4ca9017908abaa35bc2814c5b508be036105bae27338ed1422e8bb"} Nov 01 04:31:05.627509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:05.626964 6735 scope.go:122] "RemoveContainer" containerID="9921c6abaedc1fbb7e1be9470e9d5a8b51ac0cd265908d202733b41316e10f54" Nov 01 04:31:05.780975 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:05.780938 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn"] Nov 01 04:31:05.827089 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:05.827054 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2c259-7c56f575c8-vdqmn"] Nov 01 04:31:06.524509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.524474 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca23905-14e8-4101-903b-bf0631968c1f" path="/var/lib/kubelet/pods/4ca23905-14e8-4101-903b-bf0631968c1f/volumes" Nov 01 04:31:06.524869 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.524854 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b88c83-a504-44d4-8748-6ecb3200bbe1" path="/var/lib/kubelet/pods/73b88c83-a504-44d4-8748-6ecb3200bbe1/volumes" Nov 01 04:31:06.525293 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.525280 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c5ccae-f7ae-41be-bca6-decf8a9d8481" path="/var/lib/kubelet/pods/74c5ccae-f7ae-41be-bca6-decf8a9d8481/volumes" Nov 01 04:31:06.671303 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.671265 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" containerID="cri-o://72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865" gracePeriod=45 Nov 01 04:31:06.714839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.714799 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerName="main" containerID="cri-o://73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5" gracePeriod=45 Nov 01 04:31:06.737907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.737876 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv"] Nov 01 04:31:06.741071 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.741021 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c2c259-r99hv"] Nov 01 04:31:06.779204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.779124 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused" Nov 01 04:31:06.850708 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.850667 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 01 04:31:06.971574 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.971542 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:31:06.990894 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:06.990870 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 04:31:07.017058 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017002 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-dshm\") pod \"ba97abfe-c233-4de8-83d9-6c050986102d\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " Nov 01 04:31:07.017222 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017108 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-tmp-dir\") pod \"ba97abfe-c233-4de8-83d9-6c050986102d\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " Nov 01 04:31:07.017222 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017217 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-model-cache\") pod \"ba97abfe-c233-4de8-83d9-6c050986102d\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " Nov 01 04:31:07.017337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017262 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba97abfe-c233-4de8-83d9-6c050986102d-kube-api-access-v24gj\" (UniqueName: \"kubernetes.io/projected/ba97abfe-c233-4de8-83d9-6c050986102d-kube-api-access-v24gj\") pod \"ba97abfe-c233-4de8-83d9-6c050986102d\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " Nov 01 04:31:07.017337 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017299 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-home\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-home\") pod \"ba97abfe-c233-4de8-83d9-6c050986102d\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " Nov 01 04:31:07.017522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017343 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba97abfe-c233-4de8-83d9-6c050986102d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba97abfe-c233-4de8-83d9-6c050986102d-tls-certs\") pod \"ba97abfe-c233-4de8-83d9-6c050986102d\" (UID: \"ba97abfe-c233-4de8-83d9-6c050986102d\") " Nov 01 04:31:07.017522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017459 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-tmp-dir" pod "ba97abfe-c233-4de8-83d9-6c050986102d" (UID: "ba97abfe-c233-4de8-83d9-6c050986102d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.017522 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017492 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-model-cache" pod "ba97abfe-c233-4de8-83d9-6c050986102d" (UID: "ba97abfe-c233-4de8-83d9-6c050986102d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.017660 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017562 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-home" pod "ba97abfe-c233-4de8-83d9-6c050986102d" (UID: "ba97abfe-c233-4de8-83d9-6c050986102d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.017705 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017668 6735 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-tmp-dir\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.017705 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017682 6735 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-model-cache\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.017705 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.017693 6735 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-home\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.019565 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.019544 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba97abfe-c233-4de8-83d9-6c050986102d-tls-certs" pod "ba97abfe-c233-4de8-83d9-6c050986102d" (UID: "ba97abfe-c233-4de8-83d9-6c050986102d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:31:07.019640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.019590 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-dshm" pod "ba97abfe-c233-4de8-83d9-6c050986102d" (UID: "ba97abfe-c233-4de8-83d9-6c050986102d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.019640 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.019597 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba97abfe-c233-4de8-83d9-6c050986102d-kube-api-access-v24gj" pod "ba97abfe-c233-4de8-83d9-6c050986102d" (UID: "ba97abfe-c233-4de8-83d9-6c050986102d"). InnerVolumeSpecName "kube-api-access-v24gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:31:07.118405 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118319 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-tmp-dir\") pod \"639efb74-30dd-48a5-a350-f0c9210a16f6\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " Nov 01 04:31:07.118405 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118369 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/639efb74-30dd-48a5-a350-f0c9210a16f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/639efb74-30dd-48a5-a350-f0c9210a16f6-tls-certs\") pod \"639efb74-30dd-48a5-a350-f0c9210a16f6\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " Nov 01 04:31:07.118646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118417 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-home\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-home\") pod \"639efb74-30dd-48a5-a350-f0c9210a16f6\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " Nov 01 04:31:07.118646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118443 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-model-cache\") pod \"639efb74-30dd-48a5-a350-f0c9210a16f6\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " Nov 01 04:31:07.118646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118501 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639efb74-30dd-48a5-a350-f0c9210a16f6-kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/639efb74-30dd-48a5-a350-f0c9210a16f6-kube-api-access-rpcdk\") pod \"639efb74-30dd-48a5-a350-f0c9210a16f6\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " Nov 01 04:31:07.118646 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118577 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-dshm\") pod \"639efb74-30dd-48a5-a350-f0c9210a16f6\" (UID: \"639efb74-30dd-48a5-a350-f0c9210a16f6\") " Nov 01 04:31:07.118867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118627 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-tmp-dir" pod "639efb74-30dd-48a5-a350-f0c9210a16f6" (UID: "639efb74-30dd-48a5-a350-f0c9210a16f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.118867 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118722 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-home" pod "639efb74-30dd-48a5-a350-f0c9210a16f6" (UID: "639efb74-30dd-48a5-a350-f0c9210a16f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.118973 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.118927 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-model-cache" pod "639efb74-30dd-48a5-a350-f0c9210a16f6" (UID: "639efb74-30dd-48a5-a350-f0c9210a16f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.119264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.119242 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v24gj\" (UniqueName: \"kubernetes.io/projected/ba97abfe-c233-4de8-83d9-6c050986102d-kube-api-access-v24gj\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.119264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.119266 6735 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba97abfe-c233-4de8-83d9-6c050986102d-tls-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.119633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.119278 6735 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba97abfe-c233-4de8-83d9-6c050986102d-dshm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.119633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.119286 6735 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-tmp-dir\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.119633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.119293 6735 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-home\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.119633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.119301 6735 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-model-cache\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.121005 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.120984 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639efb74-30dd-48a5-a350-f0c9210a16f6-kube-api-access-rpcdk" pod "639efb74-30dd-48a5-a350-f0c9210a16f6" (UID: "639efb74-30dd-48a5-a350-f0c9210a16f6"). InnerVolumeSpecName "kube-api-access-rpcdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:31:07.121278 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.121254 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/639efb74-30dd-48a5-a350-f0c9210a16f6-tls-certs" pod "639efb74-30dd-48a5-a350-f0c9210a16f6" (UID: "639efb74-30dd-48a5-a350-f0c9210a16f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:31:07.121278 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.121271 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-dshm" pod "639efb74-30dd-48a5-a350-f0c9210a16f6" (UID: "639efb74-30dd-48a5-a350-f0c9210a16f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:31:07.219915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.219878 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcdk\" (UniqueName: \"kubernetes.io/projected/639efb74-30dd-48a5-a350-f0c9210a16f6-kube-api-access-rpcdk\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.219915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.219910 6735 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/639efb74-30dd-48a5-a350-f0c9210a16f6-dshm\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.219915 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.219920 6735 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/639efb74-30dd-48a5-a350-f0c9210a16f6-tls-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:07.642082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.642014 6735 generic.go:356] "Generic (PLEG): container finished" podID="ba97abfe-c233-4de8-83d9-6c050986102d" containerID="72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865" exitCode=0 Nov 01 04:31:07.642082 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.642084 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" Nov 01 04:31:07.642321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.642097 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" event={"ID":"ba97abfe-c233-4de8-83d9-6c050986102d","Type":"ContainerDied","Data":"72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865"} Nov 01 04:31:07.642321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.642143 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc" event={"ID":"ba97abfe-c233-4de8-83d9-6c050986102d","Type":"ContainerDied","Data":"253a34f6d9258ad2a7f5d479a8b3e9234265a318d92a61c0c13b1bcce62626e8"} Nov 01 04:31:07.642321 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.642161 6735 scope.go:122] "RemoveContainer" containerID="72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865" Nov 01 04:31:07.643568 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.643546 6735 generic.go:356] "Generic (PLEG): container finished" podID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerID="73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5" exitCode=0 Nov 01 04:31:07.643695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.643597 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" Nov 01 04:31:07.643695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.643630 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" event={"ID":"639efb74-30dd-48a5-a350-f0c9210a16f6","Type":"ContainerDied","Data":"73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5"} Nov 01 04:31:07.643695 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.643660 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9" event={"ID":"639efb74-30dd-48a5-a350-f0c9210a16f6","Type":"ContainerDied","Data":"624b6b69543d459c53148dce4a75ca0910f79836c87dc394db0e44c28c48898c"} Nov 01 04:31:07.651428 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.651410 6735 scope.go:122] "RemoveContainer" containerID="72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865" Nov 01 04:31:07.651735 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:31:07.651716 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865\": container with ID starting with 72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865 not found: ID does not exist" containerID="72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865" Nov 01 04:31:07.651796 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.651748 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865"} err="failed to get container status \"72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865\": rpc error: code = NotFound desc = could not find container \"72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865\": container with ID starting with 72bf950e3122f081d0ee05bc412451fe1d7b9ce1164b6647bc1b10b0f696e865 not found: ID does not exist" Nov 01 04:31:07.651796 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.651766 6735 scope.go:122] "RemoveContainer" containerID="73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5" Nov 01 04:31:07.660092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.660073 6735 scope.go:122] "RemoveContainer" containerID="73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5" Nov 01 04:31:07.660354 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:31:07.660334 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5\": container with ID starting with 73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5 not found: ID does not exist" containerID="73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5" Nov 01 04:31:07.660413 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.660365 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5"} err="failed to get container status \"73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5\": rpc error: code = NotFound desc = could not find container \"73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5\": container with ID starting with 73f409603b0f991adf187a4ed16c51165dca00d0e62620e9deaf825e98e674a5 not found: ID does not exist" Nov 01 04:31:07.795008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.794971 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9"] Nov 01 04:31:07.841110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.841066 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-w4-a9c5b3-kserve-6df8df868-vwkc9"] Nov 01 04:31:07.901575 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.901492 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc"] Nov 01 04:31:07.908113 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:07.908085 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a9c5b3/distinct-2-w4-a9c5b3-kserve-5bfb89ff9-st8qc"] Nov 01 04:31:08.525700 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:08.525662 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" path="/var/lib/kubelet/pods/639efb74-30dd-48a5-a350-f0c9210a16f6/volumes" Nov 01 04:31:08.526149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:08.526134 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" path="/var/lib/kubelet/pods/a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6/volumes" Nov 01 04:31:08.526509 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:08.526496 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" path="/var/lib/kubelet/pods/ba97abfe-c233-4de8-83d9-6c050986102d/volumes" Nov 01 04:31:10.990400 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:10.990363 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk"] Nov 01 04:31:10.990907 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:10.990574 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" podUID="cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" containerName="maas-api" containerID="cri-o://5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b" gracePeriod=30 Nov 01 04:31:11.240792 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.240726 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:31:11.358088 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.358050 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-openshift-service-ca\") pod \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " Nov 01 04:31:11.358267 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.358164 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-metrics-certs\") pod \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " Nov 01 04:31:11.358267 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.358219 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls\") pod \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " Nov 01 04:31:11.358267 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.358257 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-kube-api-access-jj426\" (UniqueName: \"kubernetes.io/projected/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-kube-api-access-jj426\") pod \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\" (UID: \"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6\") " Nov 01 04:31:11.358487 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.358440 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-openshift-service-ca" pod "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" (UID: "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:31:11.358631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.358614 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:11.360933 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.360907 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-metrics-certs" pod "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" (UID: "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:31:11.361080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.360972 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-kube-api-access-jj426" pod "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" (UID: "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6"). InnerVolumeSpecName "kube-api-access-jj426". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:31:11.361080 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.360982 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls" pod "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" (UID: "cc0d6f23-c9f0-4c32-9472-4553b5cf45e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:31:11.460138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.460104 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:11.460138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.460131 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:11.460138 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.460142 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj426\" (UniqueName: \"kubernetes.io/projected/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6-kube-api-access-jj426\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:31:11.671458 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.671364 6735 generic.go:356] "Generic (PLEG): container finished" podID="cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" containerID="5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b" exitCode=0 Nov 01 04:31:11.671458 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.671406 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" Nov 01 04:31:11.671706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.671460 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" event={"ID":"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6","Type":"ContainerDied","Data":"5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b"} Nov 01 04:31:11.671706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.671516 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk" event={"ID":"cc0d6f23-c9f0-4c32-9472-4553b5cf45e6","Type":"ContainerDied","Data":"9687cfac894418c853886edbf3eab7b10e9b5d1d5786ff68f5368eb3a6d48925"} Nov 01 04:31:11.671706 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.671544 6735 scope.go:122] "RemoveContainer" containerID="5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b" Nov 01 04:31:11.683194 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.683172 6735 scope.go:122] "RemoveContainer" containerID="5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b" Nov 01 04:31:11.683491 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:31:11.683472 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b\": container with ID starting with 5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b not found: ID does not exist" containerID="5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b" Nov 01 04:31:11.683553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.683500 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b"} err="failed to get container status \"5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b\": rpc error: code = NotFound desc = could not find container \"5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b\": container with ID starting with 5d56628d5cf86730dec991359636994f80ddac1d1f5a667691c6c9f0d96a2e7b not found: ID does not exist" Nov 01 04:31:11.824992 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.824956 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk"] Nov 01 04:31:11.871125 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:11.871088 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ad9e22-97c445465-gbwbk"] Nov 01 04:31:12.524188 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:12.524154 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" path="/var/lib/kubelet/pods/cc0d6f23-c9f0-4c32-9472-4553b5cf45e6/volumes" Nov 01 04:31:33.000652 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:31:33.000559 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:31:34.860634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:31:34.860586 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:32:03.001059 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:32:03.001002 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:32:05.167916 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:05.167879 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:32:14.568663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.568622 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg"] Nov 01 04:32:14.569193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.568888 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" podUID="5148b38d-b561-49b1-bce9-1a999749d468" containerName="maas-api" containerID="cri-o://9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750" gracePeriod=30 Nov 01 04:32:14.753550 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.753521 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs"] Nov 01 04:32:14.817487 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.817461 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:32:14.873214 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.873127 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5148b38d-b561-49b1-bce9-1a999749d468-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5148b38d-b561-49b1-bce9-1a999749d468-openshift-service-ca\") pod \"5148b38d-b561-49b1-bce9-1a999749d468\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " Nov 01 04:32:14.873214 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.873189 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls\") pod \"5148b38d-b561-49b1-bce9-1a999749d468\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " Nov 01 04:32:14.873451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.873267 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-metrics-certs\") pod \"5148b38d-b561-49b1-bce9-1a999749d468\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " Nov 01 04:32:14.873451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.873376 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5148b38d-b561-49b1-bce9-1a999749d468-kube-api-access-tzwqn\" (UniqueName: \"kubernetes.io/projected/5148b38d-b561-49b1-bce9-1a999749d468-kube-api-access-tzwqn\") pod \"5148b38d-b561-49b1-bce9-1a999749d468\" (UID: \"5148b38d-b561-49b1-bce9-1a999749d468\") " Nov 01 04:32:14.873451 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.873427 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5148b38d-b561-49b1-bce9-1a999749d468-openshift-service-ca" pod "5148b38d-b561-49b1-bce9-1a999749d468" (UID: "5148b38d-b561-49b1-bce9-1a999749d468"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:32:14.873681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.873665 6735 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5148b38d-b561-49b1-bce9-1a999749d468-openshift-service-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:32:14.875449 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.875428 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls" pod "5148b38d-b561-49b1-bce9-1a999749d468" (UID: "5148b38d-b561-49b1-bce9-1a999749d468"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:32:14.875553 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.875449 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-metrics-certs" pod "5148b38d-b561-49b1-bce9-1a999749d468" (UID: "5148b38d-b561-49b1-bce9-1a999749d468"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:32:14.875595 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.875547 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5148b38d-b561-49b1-bce9-1a999749d468-kube-api-access-tzwqn" pod "5148b38d-b561-49b1-bce9-1a999749d468" (UID: "5148b38d-b561-49b1-bce9-1a999749d468"). InnerVolumeSpecName "kube-api-access-tzwqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:32:14.970844 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.970804 6735 generic.go:356] "Generic (PLEG): container finished" podID="5148b38d-b561-49b1-bce9-1a999749d468" containerID="9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750" exitCode=0 Nov 01 04:32:14.971076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.970869 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" Nov 01 04:32:14.971076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.970892 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" event={"ID":"5148b38d-b561-49b1-bce9-1a999749d468","Type":"ContainerDied","Data":"9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750"} Nov 01 04:32:14.971076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.970972 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg" event={"ID":"5148b38d-b561-49b1-bce9-1a999749d468","Type":"ContainerDied","Data":"48f6877fc4d5a48011957bd14ee3d10dcfecd60b8081732b8aa19b54b0031375"} Nov 01 04:32:14.971076 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.971006 6735 scope.go:122] "RemoveContainer" containerID="9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750" Nov 01 04:32:14.974299 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.974265 6735 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-metrics-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:32:14.974299 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.974295 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzwqn\" (UniqueName: \"kubernetes.io/projected/5148b38d-b561-49b1-bce9-1a999749d468-kube-api-access-tzwqn\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:32:14.974457 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.974312 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5148b38d-b561-49b1-bce9-1a999749d468-maas-api-tls\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:32:14.986941 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.986914 6735 scope.go:122] "RemoveContainer" containerID="9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750" Nov 01 04:32:14.987520 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:32:14.987488 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750\": container with ID starting with 9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750 not found: ID does not exist" containerID="9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750" Nov 01 04:32:14.987621 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:14.987529 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750"} err="failed to get container status \"9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750\": rpc error: code = NotFound desc = could not find container \"9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750\": container with ID starting with 9778c0c87d945ec2b8de925423444b140ba1342c5a6b4f29e9a952f66a748750 not found: ID does not exist" Nov 01 04:32:15.132668 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:15.132585 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg"] Nov 01 04:32:15.137231 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:15.137208 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7545f-6dc9b49d8-htlkg"] Nov 01 04:32:16.524326 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:16.524291 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5148b38d-b561-49b1-bce9-1a999749d468" path="/var/lib/kubelet/pods/5148b38d-b561-49b1-bce9-1a999749d468/volumes" Nov 01 04:32:19.754781 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:19.754739 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" podUID="c6f24878-3d90-4f0d-97f4-01b8907714ef" containerName="payload-pre-processing" containerID="cri-o://6265293b1c688358e30403a00ae19f2480a66b3ab1493817b4aa4b5b6d2a5cbb" gracePeriod=25 Nov 01 04:32:19.997782 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:19.997749 6735 generic.go:356] "Generic (PLEG): container finished" podID="c6f24878-3d90-4f0d-97f4-01b8907714ef" containerID="6265293b1c688358e30403a00ae19f2480a66b3ab1493817b4aa4b5b6d2a5cbb" exitCode=0 Nov 01 04:32:19.997912 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:19.997813 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" event={"ID":"c6f24878-3d90-4f0d-97f4-01b8907714ef","Type":"ContainerDied","Data":"6265293b1c688358e30403a00ae19f2480a66b3ab1493817b4aa4b5b6d2a5cbb"} Nov 01 04:32:19.997912 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:19.997886 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" event={"ID":"c6f24878-3d90-4f0d-97f4-01b8907714ef","Type":"ContainerDied","Data":"98f83f4ba09115a89cf4b249ea64ed7e308395f62e76be0df652e63f65acc40f"} Nov 01 04:32:19.997912 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:19.997905 6735 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f83f4ba09115a89cf4b249ea64ed7e308395f62e76be0df652e63f65acc40f" Nov 01 04:32:20.005874 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:20.005822 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:32:20.122887 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:20.122860 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f24878-3d90-4f0d-97f4-01b8907714ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f24878-3d90-4f0d-97f4-01b8907714ef-plugins-config-volume\") pod \"c6f24878-3d90-4f0d-97f4-01b8907714ef\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " Nov 01 04:32:20.123094 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:20.123009 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f24878-3d90-4f0d-97f4-01b8907714ef-kube-api-access-44rw6\" (UniqueName: \"kubernetes.io/projected/c6f24878-3d90-4f0d-97f4-01b8907714ef-kube-api-access-44rw6\") pod \"c6f24878-3d90-4f0d-97f4-01b8907714ef\" (UID: \"c6f24878-3d90-4f0d-97f4-01b8907714ef\") " Nov 01 04:32:20.123389 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:20.123363 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f24878-3d90-4f0d-97f4-01b8907714ef-plugins-config-volume" pod "c6f24878-3d90-4f0d-97f4-01b8907714ef" (UID: "c6f24878-3d90-4f0d-97f4-01b8907714ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:32:20.125263 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:20.125237 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f24878-3d90-4f0d-97f4-01b8907714ef-kube-api-access-44rw6" pod "c6f24878-3d90-4f0d-97f4-01b8907714ef" (UID: "c6f24878-3d90-4f0d-97f4-01b8907714ef"). InnerVolumeSpecName "kube-api-access-44rw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:32:20.223945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:20.223911 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44rw6\" (UniqueName: \"kubernetes.io/projected/c6f24878-3d90-4f0d-97f4-01b8907714ef-kube-api-access-44rw6\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:32:20.223945 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:20.223945 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f24878-3d90-4f0d-97f4-01b8907714ef-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:32:21.002442 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:21.002411 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs" Nov 01 04:32:21.157008 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:21.156972 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs"] Nov 01 04:32:21.203410 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:21.203374 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f7545f-69cb5747cf-tr5hs"] Nov 01 04:32:22.525758 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:22.525721 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f24878-3d90-4f0d-97f4-01b8907714ef" path="/var/lib/kubelet/pods/c6f24878-3d90-4f0d-97f4-01b8907714ef/volumes" Nov 01 04:32:23.959665 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:23.959626 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:32:33.001156 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:32:33.001121 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:32:38.367361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:38.367330 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp"] Nov 01 04:32:43.368631 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:43.368569 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" podUID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" containerName="payload-processing" containerID="cri-o://4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1" gracePeriod=25 Nov 01 04:32:48.770191 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:48.770145 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" podUID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:32:57.345591 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:57.345554 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:32:58.769585 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:32:58.769548 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" podUID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:33:03.000576 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:33:03.000534 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:33:03.623530 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:03.623505 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:33:03.734517 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:03.734484 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-kube-api-access-6gd8v\" (UniqueName: \"kubernetes.io/projected/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-kube-api-access-6gd8v\") pod \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " Nov 01 04:33:03.734734 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:03.734583 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-plugins-config-volume\") pod \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\" (UID: \"354fb55a-c27e-40b7-a8bd-0bd9e4cab222\") " Nov 01 04:33:03.735218 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:03.735193 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-plugins-config-volume" pod "354fb55a-c27e-40b7-a8bd-0bd9e4cab222" (UID: "354fb55a-c27e-40b7-a8bd-0bd9e4cab222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:33:03.737302 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:03.737277 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-kube-api-access-6gd8v" pod "354fb55a-c27e-40b7-a8bd-0bd9e4cab222" (UID: "354fb55a-c27e-40b7-a8bd-0bd9e4cab222"). InnerVolumeSpecName "kube-api-access-6gd8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:33:03.835399 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:03.835372 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gd8v\" (UniqueName: \"kubernetes.io/projected/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-kube-api-access-6gd8v\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:03.835399 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:03.835398 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354fb55a-c27e-40b7-a8bd-0bd9e4cab222-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:04.201016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.200928 6735 generic.go:356] "Generic (PLEG): container finished" podID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" containerID="4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1" exitCode=0 Nov 01 04:33:04.201016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.200972 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" event={"ID":"354fb55a-c27e-40b7-a8bd-0bd9e4cab222","Type":"ContainerDied","Data":"4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1"} Nov 01 04:33:04.201016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.200980 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" Nov 01 04:33:04.201016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.201014 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp" event={"ID":"354fb55a-c27e-40b7-a8bd-0bd9e4cab222","Type":"ContainerDied","Data":"acf93e68eb4d140af6e1ed4d3e89844430ff2d72d0f3092e7385bb27d21d8726"} Nov 01 04:33:04.201625 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.201063 6735 scope.go:122] "RemoveContainer" containerID="4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1" Nov 01 04:33:04.210311 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.210290 6735 scope.go:122] "RemoveContainer" containerID="4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1" Nov 01 04:33:04.210557 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:33:04.210538 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1\": container with ID starting with 4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1 not found: ID does not exist" containerID="4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1" Nov 01 04:33:04.210612 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.210565 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1"} err="failed to get container status \"4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1\": rpc error: code = NotFound desc = could not find container \"4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1\": container with ID starting with 4a6760d0086058db63ca54eaee3246b7f4919812c67aba8440320faf533fe5f1 not found: ID does not exist" Nov 01 04:33:04.354361 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.354327 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp"] Nov 01 04:33:04.400471 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.400443 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4adf4c-6fdb48945c-cpmhp"] Nov 01 04:33:04.525226 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:04.525191 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" path="/var/lib/kubelet/pods/354fb55a-c27e-40b7-a8bd-0bd9e4cab222/volumes" Nov 01 04:33:09.896479 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:09.896445 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:33:09.906864 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:09.906841 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8kr8k_38b835f0-2c16-4cee-8787-9914b169aff0/ovn-acl-logging/0.log" Nov 01 04:33:10.193793 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:10.193711 6735 scope.go:122] "RemoveContainer" containerID="6265293b1c688358e30403a00ae19f2480a66b3ab1493817b4aa4b5b6d2a5cbb" Nov 01 04:33:28.926791 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:28.926753 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:33:32.335617 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:32.335574 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj"] Nov 01 04:33:32.336084 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:32.335799 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" containerID="cri-o://6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd" gracePeriod=30 Nov 01 04:33:32.478413 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:32.478377 6735 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 04:33:32.478590 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:32.478438 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 04:33:33.000954 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:33:33.000913 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:33:37.590506 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.590476 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:33:37.657672 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657580 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-certs\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.657672 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657649 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae07d12-60f6-44f9-a254-564efe3f15b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae07d12-60f6-44f9-a254-564efe3f15b5-istiod-ca-cert\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657679 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-kube-api-access-tvjms\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-kube-api-access-tvjms\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657706 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-credential-socket\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657737 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-token\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657824 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-envoy\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657862 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-socket\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657894 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-podinfo\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657952 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-data\") pod \"9ae07d12-60f6-44f9-a254-564efe3f15b5\" (UID: \"9ae07d12-60f6-44f9-a254-564efe3f15b5\") " Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.657877 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-certs" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:33:37.658099 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658067 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-credential-socket" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658199 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-socket" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658243 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae07d12-60f6-44f9-a254-564efe3f15b5-istiod-ca-cert" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658243 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-data" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658481 6735 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658496 6735 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-data\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658506 6735 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-workload-certs\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658521 6735 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae07d12-60f6-44f9-a254-564efe3f15b5-istiod-ca-cert\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.658751 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.658535 6735 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-credential-socket\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.660294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.660268 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-kube-api-access-tvjms" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "kube-api-access-tvjms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:33:37.660294 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.660286 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-envoy" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 04:33:37.660689 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.660667 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-podinfo" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 04:33:37.660689 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.660681 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-token" pod "9ae07d12-60f6-44f9-a254-564efe3f15b5" (UID: "9ae07d12-60f6-44f9-a254-564efe3f15b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:33:37.759958 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.759916 6735 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-envoy\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.759958 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.759961 6735 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-podinfo\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.760193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.759977 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjms\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-kube-api-access-tvjms\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:37.760193 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:37.759994 6735 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae07d12-60f6-44f9-a254-564efe3f15b5-istio-token\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:38.357284 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.357249 6735 generic.go:356] "Generic (PLEG): container finished" podID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerID="6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd" exitCode=0 Nov 01 04:33:38.357468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.357293 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" Nov 01 04:33:38.357468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.357341 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" event={"ID":"9ae07d12-60f6-44f9-a254-564efe3f15b5","Type":"ContainerDied","Data":"6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd"} Nov 01 04:33:38.357468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.357395 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj" event={"ID":"9ae07d12-60f6-44f9-a254-564efe3f15b5","Type":"ContainerDied","Data":"216e34550950b7f497862a1b194b3bb1954577f9fc3c02f1ddf3768f2b7432d2"} Nov 01 04:33:38.357468 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.357420 6735 scope.go:122] "RemoveContainer" containerID="6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd" Nov 01 04:33:38.366839 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.366818 6735 scope.go:122] "RemoveContainer" containerID="71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e" Nov 01 04:33:38.374931 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.374915 6735 scope.go:122] "RemoveContainer" containerID="6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd" Nov 01 04:33:38.375221 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:33:38.375202 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd\": container with ID starting with 6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd not found: ID does not exist" containerID="6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd" Nov 01 04:33:38.375290 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.375227 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd"} err="failed to get container status \"6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd\": rpc error: code = NotFound desc = could not find container \"6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd\": container with ID starting with 6386dc4551838224cb1b3cfe7df65eb881c2456954411ae598351b76f8f351fd not found: ID does not exist" Nov 01 04:33:38.375290 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.375243 6735 scope.go:122] "RemoveContainer" containerID="71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e" Nov 01 04:33:38.375464 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:33:38.375443 6735 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e\": container with ID starting with 71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e not found: ID does not exist" containerID="71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e" Nov 01 04:33:38.375501 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.375472 6735 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e"} err="failed to get container status \"71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e\": rpc error: code = NotFound desc = could not find container \"71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e\": container with ID starting with 71b5bcbd5b4450321c3b76bfc3006b927a63bba75a6e9d5c2dcd5edae29d763e not found: ID does not exist" Nov 01 04:33:38.511128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.511093 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj"] Nov 01 04:33:38.557384 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:38.557351 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fb1c1f-openshift-default-6cb7dc57c9-dqfkj"] Nov 01 04:33:40.524132 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:40.524094 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" path="/var/lib/kubelet/pods/9ae07d12-60f6-44f9-a254-564efe3f15b5/volumes" Nov 01 04:33:44.770355 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:44.770321 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb"] Nov 01 04:33:45.537682 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.537643 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4"] Nov 01 04:33:45.538043 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538021 6735 cpu_manager.go:414] "containerMap: removing container" podUID="4ca23905-14e8-4101-903b-bf0631968c1f" containerName="payload-pre-processing" Nov 01 04:33:45.538092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538052 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca23905-14e8-4101-903b-bf0631968c1f" containerName="payload-pre-processing" Nov 01 04:33:45.538092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538069 6735 cpu_manager.go:414] "containerMap: removing container" podUID="73b88c83-a504-44d4-8748-6ecb3200bbe1" containerName="payload-processing" Nov 01 04:33:45.538092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538074 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b88c83-a504-44d4-8748-6ecb3200bbe1" containerName="payload-processing" Nov 01 04:33:45.538092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538082 6735 cpu_manager.go:414] "containerMap: removing container" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerName="main" Nov 01 04:33:45.538092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538087 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerName="main" Nov 01 04:33:45.538092 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538092 6735 cpu_manager.go:414] "containerMap: removing container" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538097 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538104 6735 cpu_manager.go:414] "containerMap: removing container" podUID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" containerName="payload-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538108 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" containerName="payload-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538114 6735 cpu_manager.go:414] "containerMap: removing container" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538119 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538126 6735 cpu_manager.go:414] "containerMap: removing container" podUID="a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" containerName="revoke-keys" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538130 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" containerName="revoke-keys" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538139 6735 cpu_manager.go:414] "containerMap: removing container" podUID="5148b38d-b561-49b1-bce9-1a999749d468" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538143 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5148b38d-b561-49b1-bce9-1a999749d468" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538153 6735 cpu_manager.go:414] "containerMap: removing container" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538158 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538164 6735 cpu_manager.go:414] "containerMap: removing container" podUID="74c5ccae-f7ae-41be-bca6-decf8a9d8481" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538170 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c5ccae-f7ae-41be-bca6-decf8a9d8481" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538176 6735 cpu_manager.go:414] "containerMap: removing container" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538180 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538186 6735 cpu_manager.go:414] "containerMap: removing container" podUID="41a190b3-d99e-46f1-841c-9aa77bda2ec5" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538190 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a190b3-d99e-46f1-841c-9aa77bda2ec5" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538197 6735 cpu_manager.go:414] "containerMap: removing container" podUID="cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538201 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" containerName="maas-api" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538209 6735 cpu_manager.go:414] "containerMap: removing container" podUID="c6f24878-3d90-4f0d-97f4-01b8907714ef" containerName="payload-pre-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538213 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f24878-3d90-4f0d-97f4-01b8907714ef" containerName="payload-pre-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538219 6735 cpu_manager.go:414] "containerMap: removing container" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538222 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538231 6735 cpu_manager.go:414] "containerMap: removing container" podUID="78b560bf-0e29-416b-9f70-00223c46a8d1" containerName="payload-pre-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538235 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b560bf-0e29-416b-9f70-00223c46a8d1" containerName="payload-pre-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538244 6735 cpu_manager.go:414] "containerMap: removing container" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538249 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538257 6735 cpu_manager.go:414] "containerMap: removing container" podUID="01ecd83a-560d-4e93-81cb-3474b8d9881b" containerName="payload-pre-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538261 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ecd83a-560d-4e93-81cb-3474b8d9881b" containerName="payload-pre-processing" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538268 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538272 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538283 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538287 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.538280 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538295 6735 cpu_manager.go:414] "containerMap: removing container" podUID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" containerName="payload-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538299 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" containerName="payload-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538305 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538309 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538358 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="97aa7198-e7c4-43f4-a101-f80f82bf3f70" containerName="payload-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538366 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538372 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="a983147c-7cfb-4f6c-8c8f-6ff708f1cbe6" containerName="revoke-keys" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538378 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="5148b38d-b561-49b1-bce9-1a999749d468" containerName="maas-api" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538385 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c5ccae-f7ae-41be-bca6-decf8a9d8481" containerName="maas-api" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538391 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba97abfe-c233-4de8-83d9-6c050986102d" containerName="main" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538396 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a190b3-d99e-46f1-841c-9aa77bda2ec5" containerName="maas-api" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538401 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0d6f23-c9f0-4c32-9472-4553b5cf45e6" containerName="maas-api" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538407 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f24878-3d90-4f0d-97f4-01b8907714ef" containerName="payload-pre-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538413 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538418 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b560bf-0e29-416b-9f70-00223c46a8d1" containerName="payload-pre-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538424 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538429 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ecd83a-560d-4e93-81cb-3474b8d9881b" containerName="payload-pre-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538433 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538439 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538444 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="354fb55a-c27e-40b7-a8bd-0bd9e4cab222" containerName="payload-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538448 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e44b907-7cec-40a8-87f4-3f697a247d25" containerName="cleanup" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538452 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca23905-14e8-4101-903b-bf0631968c1f" containerName="payload-pre-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538457 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b88c83-a504-44d4-8748-6ecb3200bbe1" containerName="payload-processing" Nov 01 04:33:45.539354 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.538461 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="639efb74-30dd-48a5-a350-f0c9210a16f6" containerName="main" Nov 01 04:33:45.543300 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.543277 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.596602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.596564 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4"] Nov 01 04:33:45.632006 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.631979 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 04:33:45.632182 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.632152 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ggdg\"" Nov 01 04:33:45.633549 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.633530 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.633602 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.633565 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.633745 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.633722 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6d2\" (UniqueName: \"kubernetes.io/projected/1144231c-f590-428e-9bde-3bfa60df17f5-kube-api-access-tp6d2\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.734926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.734887 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6d2\" (UniqueName: \"kubernetes.io/projected/1144231c-f590-428e-9bde-3bfa60df17f5-kube-api-access-tp6d2\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.735105 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.734937 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.735105 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.734990 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.735681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.735657 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.737581 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.737558 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.792358 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.792292 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6d2\" (UniqueName: \"kubernetes.io/projected/1144231c-f590-428e-9bde-3bfa60df17f5-kube-api-access-tp6d2\") pod \"maas-api-revoke-sub-89510534-ljff4\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:45.860264 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:45.860228 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:46.224554 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:46.224454 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4"] Nov 01 04:33:46.227727 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:33:46.227700 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1144231c_f590_428e_9bde_3bfa60df17f5.slice/crio-82bb9067302c3202dad5c9751dc3ef3d607b39b27c59e8fe730ab4bddc24602f WatchSource:0}: Error finding container 82bb9067302c3202dad5c9751dc3ef3d607b39b27c59e8fe730ab4bddc24602f: Status 404 returned error can't find the container with id 82bb9067302c3202dad5c9751dc3ef3d607b39b27c59e8fe730ab4bddc24602f Nov 01 04:33:46.397584 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:46.397551 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" event={"ID":"1144231c-f590-428e-9bde-3bfa60df17f5","Type":"ContainerStarted","Data":"82bb9067302c3202dad5c9751dc3ef3d607b39b27c59e8fe730ab4bddc24602f"} Nov 01 04:33:47.404204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:47.404098 6735 generic.go:356] "Generic (PLEG): container finished" podID="1144231c-f590-428e-9bde-3bfa60df17f5" containerID="8f16a118c9d3839aaa4a9516c9c7476894d97cd44602939a54242c01addc7e49" exitCode=0 Nov 01 04:33:47.404204 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:47.404186 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" event={"ID":"1144231c-f590-428e-9bde-3bfa60df17f5","Type":"ContainerDied","Data":"8f16a118c9d3839aaa4a9516c9c7476894d97cd44602939a54242c01addc7e49"} Nov 01 04:33:48.543079 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.543029 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:48.664982 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.664898 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1144231c-f590-428e-9bde-3bfa60df17f5-kube-api-access-tp6d2\" (UniqueName: \"kubernetes.io/projected/1144231c-f590-428e-9bde-3bfa60df17f5-kube-api-access-tp6d2\") pod \"1144231c-f590-428e-9bde-3bfa60df17f5\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " Nov 01 04:33:48.665173 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.665029 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-xks-ca\") pod \"1144231c-f590-428e-9bde-3bfa60df17f5\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " Nov 01 04:33:48.665173 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.665078 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-openshift-ca\") pod \"1144231c-f590-428e-9bde-3bfa60df17f5\" (UID: \"1144231c-f590-428e-9bde-3bfa60df17f5\") " Nov 01 04:33:48.665439 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.665419 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-openshift-ca" pod "1144231c-f590-428e-9bde-3bfa60df17f5" (UID: "1144231c-f590-428e-9bde-3bfa60df17f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:33:48.667142 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.667120 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1144231c-f590-428e-9bde-3bfa60df17f5-kube-api-access-tp6d2" pod "1144231c-f590-428e-9bde-3bfa60df17f5" (UID: "1144231c-f590-428e-9bde-3bfa60df17f5"). InnerVolumeSpecName "kube-api-access-tp6d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:33:48.667241 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.667225 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-xks-ca" pod "1144231c-f590-428e-9bde-3bfa60df17f5" (UID: "1144231c-f590-428e-9bde-3bfa60df17f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 04:33:48.766626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.766587 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-xks-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:48.766626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.766618 6735 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1144231c-f590-428e-9bde-3bfa60df17f5-maas-api-openshift-ca\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:48.766626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:48.766635 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp6d2\" (UniqueName: \"kubernetes.io/projected/1144231c-f590-428e-9bde-3bfa60df17f5-kube-api-access-tp6d2\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:33:49.415367 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:49.415339 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" Nov 01 04:33:49.415567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:49.415331 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89510534-ljff4" event={"ID":"1144231c-f590-428e-9bde-3bfa60df17f5","Type":"ContainerDied","Data":"82bb9067302c3202dad5c9751dc3ef3d607b39b27c59e8fe730ab4bddc24602f"} Nov 01 04:33:49.415567 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:49.415466 6735 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82bb9067302c3202dad5c9751dc3ef3d607b39b27c59e8fe730ab4bddc24602f" Nov 01 04:33:49.771600 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:49.771551 6735 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" podUID="18525227-695f-424a-80dd-9b2e56509cb4" containerName="payload-processing" containerID="cri-o://b2ac37304f78db1f5bae6d56173f8bb36dc13e8fb36c32c4726045937ad3a300" gracePeriod=25 Nov 01 04:33:53.619110 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:33:53.619065 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" podUID="18525227-695f-424a-80dd-9b2e56509cb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:34:02.346785 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:02.346752 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:34:03.000582 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:34:03.000548 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:34:03.618965 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:03.618924 6735 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" podUID="18525227-695f-424a-80dd-9b2e56509cb4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 04:34:10.041666 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.041593 6735 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 04:34:10.158679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.158272 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18525227-695f-424a-80dd-9b2e56509cb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18525227-695f-424a-80dd-9b2e56509cb4-plugins-config-volume\") pod \"18525227-695f-424a-80dd-9b2e56509cb4\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " Nov 01 04:34:10.158679 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.158325 6735 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18525227-695f-424a-80dd-9b2e56509cb4-kube-api-access-w45kt\" (UniqueName: \"kubernetes.io/projected/18525227-695f-424a-80dd-9b2e56509cb4-kube-api-access-w45kt\") pod \"18525227-695f-424a-80dd-9b2e56509cb4\" (UID: \"18525227-695f-424a-80dd-9b2e56509cb4\") " Nov 01 04:34:10.159158 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.159067 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18525227-695f-424a-80dd-9b2e56509cb4-plugins-config-volume" pod "18525227-695f-424a-80dd-9b2e56509cb4" (UID: "18525227-695f-424a-80dd-9b2e56509cb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 04:34:10.161146 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.161117 6735 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18525227-695f-424a-80dd-9b2e56509cb4-kube-api-access-w45kt" pod "18525227-695f-424a-80dd-9b2e56509cb4" (UID: "18525227-695f-424a-80dd-9b2e56509cb4"). InnerVolumeSpecName "kube-api-access-w45kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 04:34:10.219756 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.219725 6735 scope.go:122] "RemoveContainer" containerID="b2ac37304f78db1f5bae6d56173f8bb36dc13e8fb36c32c4726045937ad3a300" Nov 01 04:34:10.262569 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.262275 6735 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45kt\" (UniqueName: \"kubernetes.io/projected/18525227-695f-424a-80dd-9b2e56509cb4-kube-api-access-w45kt\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:34:10.262569 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.262307 6735 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18525227-695f-424a-80dd-9b2e56509cb4-plugins-config-volume\") on node \"ip-10-0-128-69.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 04:34:10.506454 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.506419 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" Nov 01 04:34:10.506626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.506419 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" event={"ID":"18525227-695f-424a-80dd-9b2e56509cb4","Type":"ContainerDied","Data":"b2ac37304f78db1f5bae6d56173f8bb36dc13e8fb36c32c4726045937ad3a300"} Nov 01 04:34:10.506626 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.506564 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb" event={"ID":"18525227-695f-424a-80dd-9b2e56509cb4","Type":"ContainerDied","Data":"f19ce1d4676a52d644f01c578c7d748addc9c3f2284a386895bc6ba11d004340"} Nov 01 04:34:10.720327 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.720288 6735 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb"] Nov 01 04:34:10.724651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:10.724621 6735 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-865066-b699f8cbf-q9ghb"] Nov 01 04:34:12.524809 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:12.524774 6735 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18525227-695f-424a-80dd-9b2e56509cb4" path="/var/lib/kubelet/pods/18525227-695f-424a-80dd-9b2e56509cb4/volumes" Nov 01 04:34:13.449095 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:13.449057 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4nvf6_aec50471-69e8-4b0e-807a-3d70e92c408e/limitador/0.log" Nov 01 04:34:14.447645 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:14.447617 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bdf4bddbc-hwvmh_f4ad6801-8f5a-4405-96e1-da3ee74cccdb/manager/0.log" Nov 01 04:34:17.236016 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:17.235980 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4nvf6"] Nov 01 04:34:27.264289 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:27.264249 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bdf4bddbc-hwvmh_f4ad6801-8f5a-4405-96e1-da3ee74cccdb/manager/0.log" Nov 01 04:34:30.184964 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:30.184937 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77c84bfd5-s2mth_8bcb2e9a-1416-4711-b3af-94b60d002b15/maas-api/0.log" Nov 01 04:34:30.844477 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:30.844447 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-89510534-ljff4_1144231c-f590-428e-9bde-3bfa60df17f5/revoke-keys/0.log" Nov 01 04:34:32.999804 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:34:32.999768 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:34:35.112685 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:35.112654 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4nvf6_aec50471-69e8-4b0e-807a-3d70e92c408e/limitador/0.log" Nov 01 04:34:36.326191 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:36.326157 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8lmz6_55dbeac6-535c-4fbc-a6e1-3bf88447cd4e/discovery/0.log" Nov 01 04:34:36.771633 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:36.771606 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f668977d-wjqtd_a77e7502-9975-4ff7-b0e1-687e12a045f8/kube-auth-proxy/0.log" Nov 01 04:34:36.984926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:36.984899 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p79qx_bd9b484a-a7c6-438e-b741-13e153e3f2ff/istio-proxy/0.log" Nov 01 04:34:37.412166 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:37.412133 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sdhwx_7715e8b4-ae01-4678-a074-0409f721d0d0/payload-pre-processing/0.log" Nov 01 04:34:37.627878 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:37.627846 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4qdkd_6dc0dfac-2281-4847-a3fe-2d11976315cc/payload-processing/0.log" Nov 01 04:34:38.616981 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:38.616951 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9z46n_98e51262-5076-48d9-878f-68948911c3b4/main/0.log" Nov 01 04:34:38.817339 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:38.817301 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-22fpx_a683cab9-8bf5-48e5-9f20-e4d26754b48e/main/0.log" Nov 01 04:34:39.470710 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:39.470678 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86d889_ee8a97ea-a619-4e99-b374-bbb6dc85177a/main/0.log" Nov 01 04:34:39.905242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:34:39.905161 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-54rmh_39329c16-0f40-4151-821a-6aac58f565bd/main/0.log" Nov 01 04:35:01.162061 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:01.162003 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bdf4bddbc-hwvmh_f4ad6801-8f5a-4405-96e1-da3ee74cccdb/manager/0.log" Nov 01 04:35:02.647576 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:02.647541 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nwbf4_128c7b38-4784-403c-be55-e2f1561c2f72/global-pull-secret-syncer/0.log" Nov 01 04:35:03.002660 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:35:03.002622 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor" Nov 01 04:35:05.254045 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:05.254001 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k2s25_c1bed51d-4e8a-4f9c-8c9b-d0aaf0691216/konnectivity-agent/0.log" Nov 01 04:35:06.016342 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:06.016308 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-69.us-west-2.compute.internal_63ad88aaffc4eb2dd4273e995e15f57f/haproxy/0.log" Nov 01 04:35:06.206149 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:06.206098 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bdf4bddbc-hwvmh_f4ad6801-8f5a-4405-96e1-da3ee74cccdb/manager/0.log" Nov 01 04:35:10.694634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:10.694596 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bdf4bddbc-hwvmh_f4ad6801-8f5a-4405-96e1-da3ee74cccdb/manager/0.log" Nov 01 04:35:12.975555 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.975522 6735 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp"] Nov 01 04:35:12.975953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.975897 6735 cpu_manager.go:414] "containerMap: removing container" podUID="1144231c-f590-428e-9bde-3bfa60df17f5" containerName="revoke-keys" Nov 01 04:35:12.975953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.975909 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1144231c-f590-428e-9bde-3bfa60df17f5" containerName="revoke-keys" Nov 01 04:35:12.975953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.975922 6735 cpu_manager.go:414] "containerMap: removing container" podUID="18525227-695f-424a-80dd-9b2e56509cb4" containerName="payload-processing" Nov 01 04:35:12.975953 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.975928 6735 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18525227-695f-424a-80dd-9b2e56509cb4" containerName="payload-processing" Nov 01 04:35:12.976128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.975984 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="1144231c-f590-428e-9bde-3bfa60df17f5" containerName="revoke-keys" Nov 01 04:35:12.976128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.975992 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="18525227-695f-424a-80dd-9b2e56509cb4" containerName="payload-processing" Nov 01 04:35:12.976128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.976000 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae07d12-60f6-44f9-a254-564efe3f15b5" containerName="istio-proxy" Nov 01 04:35:12.976128 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.976010 6735 memory_manager.go:365] "RemoveStaleState removing state" podUID="401bdad5-fe06-4bf6-b145-5ae0a9af9e90" containerName="cleanup" Nov 01 04:35:12.979210 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:12.979190 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.028536 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.028511 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jth2x\"/\"openshift-service-ca.crt\"" Nov 01 04:35:13.057926 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.057899 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-sys\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.058123 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.057952 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-podres\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.058123 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.057970 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-proc\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.058242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.058117 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59w8m\" (UniqueName: \"kubernetes.io/projected/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-kube-api-access-59w8m\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.058242 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.058164 6735 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-lib-modules\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.074914 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.074891 6735 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jth2x\"/\"kube-root-ca.crt\"" Nov 01 04:35:13.075168 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.075150 6735 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jth2x\"/\"default-dockercfg-j7x5g\"" Nov 01 04:35:13.076320 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.076300 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp"] Nov 01 04:35:13.159647 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159615 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-podres\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159647 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159644 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-proc\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159665 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59w8m\" (UniqueName: \"kubernetes.io/projected/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-kube-api-access-59w8m\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159697 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-lib-modules\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159746 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-podres\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159761 6735 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-sys\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159749 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-proc\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159809 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-sys\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.159871 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.159796 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-lib-modules\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.217282 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.217251 6735 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59w8m\" (UniqueName: \"kubernetes.io/projected/fd4d3cf3-de9b-453a-b514-2131dd2d0a0d-kube-api-access-59w8m\") pod \"perf-node-gather-daemonset-2tzhp\" (UID: \"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d\") " pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.290994 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.290918 6735 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.653875 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.653794 6735 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp"] Nov 01 04:35:13.656932 ip-10-0-128-69 kubenswrapper[6735]: W1101 04:35:13.656901 6735 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfd4d3cf3_de9b_453a_b514_2131dd2d0a0d.slice/crio-a53fa59e68a608e0e8717b76eb88a7dfbaf1e6eebc8258477449df3c6f5918bb WatchSource:0}: Error finding container a53fa59e68a608e0e8717b76eb88a7dfbaf1e6eebc8258477449df3c6f5918bb: Status 404 returned error can't find the container with id a53fa59e68a608e0e8717b76eb88a7dfbaf1e6eebc8258477449df3c6f5918bb Nov 01 04:35:13.780306 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.780274 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" event={"ID":"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d","Type":"ContainerStarted","Data":"238c4ce64b17b065f2bb1915a2ca46a460bea6a8f7ff8afd0bbfb94d12090d3b"} Nov 01 04:35:13.780432 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.780327 6735 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" event={"ID":"fd4d3cf3-de9b-453a-b514-2131dd2d0a0d","Type":"ContainerStarted","Data":"a53fa59e68a608e0e8717b76eb88a7dfbaf1e6eebc8258477449df3c6f5918bb"} Nov 01 04:35:13.780432 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.780357 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:13.889886 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:13.889834 6735 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" podStartSLOduration=1.8898202670000002 podStartE2EDuration="1.889820267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 04:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:35:13.889556375 +0000 UTC m=+3125.478651005" watchObservedRunningTime="2026-11-01 04:35:13.889820267 +0000 UTC m=+3125.478914896" Nov 01 04:35:14.891681 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:14.891652 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4nvf6_aec50471-69e8-4b0e-807a-3d70e92c408e/limitador/0.log" Nov 01 04:35:19.798124 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:19.798066 6735 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jth2x/perf-node-gather-daemonset-2tzhp" Nov 01 04:35:20.390780 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:20.390745 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-8lmz6_55dbeac6-535c-4fbc-a6e1-3bf88447cd4e/discovery/0.log" Nov 01 04:35:20.705554 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:20.705470 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f668977d-wjqtd_a77e7502-9975-4ff7-b0e1-687e12a045f8/kube-auth-proxy/0.log" Nov 01 04:35:20.922892 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:20.922859 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-p79qx_bd9b484a-a7c6-438e-b741-13e153e3f2ff/istio-proxy/0.log" Nov 01 04:35:20.998177 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:20.998150 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-75rhs_34608ae4-4158-4e76-88bd-bdb4d1381f87/kube-state-metrics/0.log" Nov 01 04:35:21.347651 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:21.347552 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sdhwx_7715e8b4-ae01-4678-a074-0409f721d0d0/payload-pre-processing/0.log" Nov 01 04:35:21.386621 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:21.386596 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-75rhs_34608ae4-4158-4e76-88bd-bdb4d1381f87/kube-rbac-proxy-main/0.log" Nov 01 04:35:21.559964 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:21.559929 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-4qdkd_6dc0dfac-2281-4847-a3fe-2d11976315cc/payload-processing/0.log" Nov 01 04:35:21.784019 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:21.783992 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-75rhs_34608ae4-4158-4e76-88bd-bdb4d1381f87/kube-rbac-proxy-self/0.log" Nov 01 04:35:24.179908 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:24.179878 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4gkc_07e58839-259c-453a-98f1-dac051fb8f9e/node-exporter/0.log" Nov 01 04:35:24.584976 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:24.584942 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4gkc_07e58839-259c-453a-98f1-dac051fb8f9e/kube-rbac-proxy/0.log" Nov 01 04:35:24.989663 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:24.989638 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4gkc_07e58839-259c-453a-98f1-dac051fb8f9e/init-textfile/0.log" Nov 01 04:35:26.590634 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:26.590592 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-szlw8_5cf75ba2-0374-460d-af57-81cebb276672/kube-rbac-proxy-main/0.log" Nov 01 04:35:26.991648 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:26.991621 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-szlw8_5cf75ba2-0374-460d-af57-81cebb276672/kube-rbac-proxy-self/0.log" Nov 01 04:35:27.388743 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:27.388667 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-szlw8_5cf75ba2-0374-460d-af57-81cebb276672/openshift-state-metrics/0.log" Nov 01 04:35:28.027432 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:28.027400 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4nvf6_aec50471-69e8-4b0e-807a-3d70e92c408e/limitador/0.log" Nov 01 04:35:30.637157 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:30.637130 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7rlnf_6f1e0e21-7af3-4316-b343-bcc39d1dcb85/prometheus-operator/0.log" Nov 01 04:35:31.027937 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:31.027909 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7rlnf_6f1e0e21-7af3-4316-b343-bcc39d1dcb85/kube-rbac-proxy/0.log" Nov 01 04:35:31.827810 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:31.827765 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7456b8b6fc-lzzzt_ab862706-1c0c-4324-86d7-59c9b6313672/telemeter-client/0.log" Nov 01 04:35:32.225240 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:32.225203 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7456b8b6fc-lzzzt_ab862706-1c0c-4324-86d7-59c9b6313672/reload/0.log" Nov 01 04:35:32.623899 ip-10-0-128-69 kubenswrapper[6735]: I1101 04:35:32.623820 6735 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7456b8b6fc-lzzzt_ab862706-1c0c-4324-86d7-59c9b6313672/kube-rbac-proxy/0.log" Nov 01 04:35:33.000513 ip-10-0-128-69 kubenswrapper[6735]: E1101 04:35:33.000471 6735 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lzs95" containerName="extractor"